In a visual search task in which a specific target symbol is sought from a number of alternatives, one can find the target symbol efficiently in a relatively short time. Hick's law states that the required time for such a task is proportional to the logarithm of the number of symbols. In order to understand the mechanism for such an efficient visual search, we proposed a hypothesis that the trajectory comprising the sequential gaze positions within the search area establishes a small-world-network-like structure that contributes to reducing the search time. To confirm this hypothesis, we first measured the gaze position dynamics from eleven participants during the visual search task, and showed that the frequency distribution of the distances between successive gaze positions obeys a power law with an exponent of 2.4, and that the distance time series exhibited a long-term memory characterized by a Hurst exponent of 0.7. Based on these results, we proposed a simple model that can reproduce a time series of distances. Numerical simulations of the model indicated that the search time was minimized when the exponent of the power law was close to the experimental value, suggesting that eye movement during the visual search task is optimized so as to reduce the search time.