Music often moves us and provides great pleasure. The origins of this ability of music remain a mystery. Recently, an increasing number of studies have proposed evolutionary theories of human musicality, although several researchers deny the adaptive value of music. In this paper, the origins of the pleasure that music provides were discussed in terms of human evolution and cultural adaptation. A possibility in which both evolutionary and cultural adaptation resulted in the pleasure of music was shown. Next, the relationship between music and visual stimuli was focused on. Owing to the development of music devices and the Internet, music is heard increasingly with visual stimuli, like background music in everyday life, films, drama, dance, computer games, music videos, etc. Based on the author's findings on the cross-modal effects between music and visual stimuli, the pleasure of listening to music with visual stimuli was discussed. It was emphasized that the meaning of music is strengthened, changed, and multi-layered by visual stimuli when it is enjoyed with the visual stimuli.