The population size, i.e., the number of candidate solutions per iteration, is the only parameter for the covariance matrix adaptation evolution strategy (CMA-ES) that needs to be tuned depending on the ruggedness and the uncertainty of the objective function. The population size has a great impact on the performance of the CMA-ES, however, it is prohibitively expensive in black-box scenario to tune the population size in advance. Moreover, a reasonable population size is not constant during the optimization. In this paper, we propose a novel strategy to adapt the population size. We introduce the evolution path in the parameter space of the Gaussian distribution, which accumulates successive parameter updates. Based on the length of the evolution path with respect to the Fisher metric, we quantify the accuracy of the parameter update. The population size is then updated so that the quantified accuracy is kept in the constant range during search. The proposed strategy is evaluated on test functions including rugged functions and noisy functions where a larger population size is known to help to find a better solution. The experimental results show that the population size is kept as small as the default population size on unimodal functions, and it is increased at the early stage of the optimization of multimodal functions and decreased after the sampling distribution is concentrated in a single valley of a local optimum. On noisy test functions, the proposed strategy start increasing the population size when the noise-to-signal ratio becomes relatively high. The proposed strategy is compared with the CMA-ES and the state-of-the-art uncertainty handling in the CMA-ES, namely UH-CMA-ES, with a hand-tuned population sizes.