Previous studies have shown that color preferences can be modulated by altering the observer’s mental set about color through environmental change (e.g., seasonal change). This study investigated whether color preferences changed when the mental set on color was manipulated, by providing participants with instructions about different cultural sets without any physical change in the environment. To this end, we compared the color preferences of Japanese participants under three different cultural sets, namely, “colors that are used in America,” “colors of Japanese traditional culture,” or no specified cultural set. We also investigated the degree to which participants matched colors with each cultural set, i.e., American or traditional Japanese. The results demonstrated the modulation of color preferences. Furthermore, the size of the modulation for each color positively correlated with the degree of the match between each color and each cultural set, suggesting that the cultural set affected the color preferences.