To effectively design tactile textures of product surfaces, it is essential to specify the semantically multilayered and multidimensional structure of human perceptual, emotional, and preferential expressions pertaining to touching materials. We implemented a method to identify individual differences in the structure of these expressions, based on subjective reports of the mutual impacts among 29 adjective dyads used to describe tactile experiences of material samples. Results showed that the 11 university students who responded to 46 types of flat materials were clustered into three statistically different groups. The structures were largely composed of three layers of expressions: psychophysical (bottom), emotional or material attributes (middle), and preferential (top). All groups selected the same adjective dyads encompassed in the bottom layer, relating to percepts of physical quantities, which indicates the similarity of psychophysical percepts among the participants; however, the middle and top layers significantly differed across individuals. These results indicate that under the tested conditions, there are individual differences in the semantic structures of emotional and preferential experiences; however, such differences in psychophysical expressions are relatively minor.