Primates behave differently toward infants and adults and experimental studies have also confirmed that they visually differentiate between adults and infants. Infants have some visual characteristics that primates may potentially use as age cues. For example, infants in some species have special coat or skin colors that are different from adults. Such coloration is called “infantile coloration” and the extent of it varies across species. Although previous studies have suggested several possible functional roles of it, such as inducing alloparenting behavior, they still remain unclear. Contrary to such species-specific color features, infants also have morphological characteristics that exist across species. “Baby schema” is a set of infantile morphological features, such as bigger eyes, smaller nose and mouth, proposed by Konrad Lorenz. Baby schema is supposed to be shared with various species and works as a releaser of caretaking behavior. I reviewed the studies investigating the effects of infantile coloration and baby schema on behaviors and cognition. Although studies showed that humans show robust preference for baby schema, there has been so far no evidence suggesting that non-human primates are also sensitive to it. In contrast, there is some evidence indicating the importance of species-specific infantile features, such as infantile coloration, for non-human primates. Thus, it may be possible that preference for baby schema was specifically acquired by humans during evolution, while species-specific infantile features, like infantile coloration, are more relevant for non-human primates. Further investigation combining studies with different approaches, such as observational studies in the field, comparative cognitive studies in labs, and image analysis of various species, will help us to understand the evolution of infantile features of each primate species.