While surrounded by a built environment, various interpersonal cognitions and behaviors such as impression formation, communication, and cooperation are regulated and executed. It has become clear through many years of environmental psychology studies that the physical environment not only surrounds but also implicitly influences interpersonal cognition and behavior. This paper reviews the literature on the effects of visual environmental factors specifically brightness, color, and depth on interpersonal cognition and behavior in a little more than 20 years, clarifying the achievements and expansion of the research area, the challenges to be overcome, and the outlooks for the future directions.