The presence of a peer passenger psychologically affects a driver’s behavior, which may be considered a risk factor for a vehicle accident. Although previous studies have focused on the peer-passenger effect among middle-aged drivers, little is known about such effect both in middle-aged and young drivers. This study explored the factor structure of the peer-passenger effect and determinants of this effect in both middle-aged and young drivers. A total of 282 drivers completed a self-report questionnaire. Factor analysis revealed five factors of the peer-passenger effect : dissatisfaction with advice, self-uneasiness, security consciousness, and calmness and silence in car. Multiple regression analyses were conducted to examine how strongly the five demographic variables of gender, age, a license career, and frequencies of driving and of a presence of peer passenger could influence each factor. Results suggested that a middle-aged driver became irritated more easily than a young driver, when receiving unnecessary advice from a peer passenger. Younger drivers concentrated on driving so much that they could not afford to pay attention to peer passenger. Furthermore, the frequency of the presence of a peer passenger could promote the peer-passenger effect in both young and middle-aged drivers.