People are generally unable to correctly determine the fixation point from the gaze direction when facing another person. This study investigated this tendency from the viewpoint of the anisotropy of space. Experiment 1 showed that compared with the physical space, the gaze perceptional space was 1.227 times larger laterally and 0.516 times narrower sagittally, suggesting that the gaze perceptional space has an orthotropic property. Furthermore, this space had an anisotropic property in the oblique direction. In Experiment 2, the space was constructed from the verbal cues of distance and angle, with the same size as the physical space in Experiment 1. Compared with the physical space, the space constructed from the verbal cues was 0.866 times larger laterally and 0.783 times narrower sagittally. These results show that the gaze perceptional space differs from the space constructed by verbal cues in terms of the degree of anisotropy. They also suggest that gaze direction was not judged on the basis of quantitative verbal representation concerning distance and angle.