This study attempted to investigate the differences in tourists’ perceptions of destination images based on their past travel experiences. Tourism research suggests that past travel experiences affect present tourist behavior in terms of travel career and number of visits. In this study, we consider the tourist as a person seeking mastery through travel experiences and use measures of guest experience from 47 prefectures in Japan. Five hundred respondents living in Osaka were asked to complete a questionnaire via the Internet. Respondents had to identify 10 destination images and write about their travel experiences in as many of the 47 prefectures as they had visited. The main results were as follows: (1) respondents were classified into four clusters according to their travel experiences, (2) tourists who had traveled to more prefectures had an image of the destination based on geographical location and destination characteristics, and (3) tourists who had previously visited a particular destination and had been to more prefectures had a clear image of the destination. Based on these results, the process through which tourists develop expertise was discussed.