【Purpose】In this study, we investigated the physical activity and lifestyle factors that affect physical fitness in infants. 【Methods】We investigated the physical fitness, physical activity, and lifestyle, including the home environment and exercise orientation, in 194 infants at three facilities, by employing a covariance structure analysis model.【Results】The results of the study clarified that the factors and mechanisms that affect physical fitness differed between the boys and the girls. In the boys, "physical activity," "exercise orientation," and "home exercise environment" showed a direct impact on physical fitness. However, in the girls, only "physical activity" showed a direct effect. Furthermore, for the girls, the "home exercise environment" affected the "exercise orientation," such as the opportunity for parents and children to exercise together, and the "exercise orientation" also affected "physical activity." In terms of physical activity, moderateto vigorous-intensity physical activity and PAL in girls, and steps and PAL in boys were factors that affected physical fitness. In the boys'group, it was revealed that the play time had a negative effect on physical fitness.