Whether engaging in costly punishment can be an adaptive strategy by enhancing the punisher’s reputation is a crucial question in efforts to explain punishment behaviors. However, empirical findings on this question are mixed. Based on Raihani and Bshary’s (2015) argument, the current study empirically examined the possibility that how a punisher’s motives are estimated affects the reputation of the punisher. We employed a vignette experiment which was designed to make it simple for respondents to estimate each punisher’s motives. Each vignette described a defector in a social dilemma and a punisher who punished the defector using one of four types of punishment. Respondents then estimated the punisher’s motives and evaluated their impressions. The results revealed that the estimated motives depended on the type of punishment which the punishers utilized and that evaluations of punishers varied across the four types of punishment. Thus, the context in which punishment is employed may affect a punisher’s reputation, and this in turn might ultimately determine whether engaging in punishment is adaptive.