A speech-driven embodied listening system with mirroring of speaker’s pupil response

Translated Abstract

Listening is an important synchronous behavior in building friendly relationship with others. Among them, mirroring is a technique that creates a sense of unity by imitating behaviors. However, excessive imitation may damage relations and trust, because imitating behaviors are voluntary movements that include intentions. Therefore, unconscious behaviors are desirable for the mirroring. We focused on the pupil response as unconscious behaviors and analyzed the relationship between the speaker’s speech and his/her own pupil response and confirmed that the pupil dilates in synchronization with the speech. In addition, a pupil response robot that expresses various pupil responses was developed, and the effectiveness of the robot was demonstrated. These findings suggest that controlling pupil response of the robot can enhance relations and trust in human-robot interaction. Therefore, an interaction design with active listening is expected by incorporating a mirrored pupil response into social robots. In this study, we developed a speech-driven embodied listening system with mirroring of the speaker’s pupil response. This system realizes a positive listener’s attitude by generating a pupil response as well as a nodding response based on speech input. The effectiveness of the system was demonstrated by sensory evaluations.

References

• [1]  LOVOT: GROOVE X, Inc., (https://lovot.life/) (accessed 2024-1-10) (in Japanese).
• [2]  aibo: Sony Group Corporation, (https://aibo.sony.jp/) (accessed 2024-1-10) (in Japanese).
• [3]  Ozaki, F.: The current status of elderly care robots, Japanese Journal of Geriatrics, Vol.57, No.3, pp.224–235 (2020) (in Japanese).
• [4]  Rogers, C. R. and Wallen, J. L.: Counseling with returned servicemen, McGraw-Hill Book Company, (1946).
• [5]  Nozawa, T.: Communication Captured by Multimodal Synchronized Sensing, Jpn. J. Appl. Phys., Vol.89, No.12, pp.734–736 (2020). (in Japanese).
• [6]  Uchida, S., Terasaka, A. and Ikeda, K.: The influence of mirroring in dialogic situation on interpersonal attraction, The Journal of Human Development and Clinical Psychology, Vol.24, pp.9–16 (2018) (in Japanese).
• [7]  Nagai, Y.: From Understanding to Assisting: Cognitive Mirroring that Makes Developmental Disorders Observable, Journal of Human Interface Society, Vol.21, No.1, pp.5–10 (2019) (in Japanese).
• [8]  Senzoku, S., Sakai, H. and Usui, S.: Mathematical model for pupillary light reflex system and its validity, IEICE technical report, Vol.102, No.731, pp.155–160 (2003) (in Japanese).
• [9]  Asano, S., Nakayama, M. and Shimizu, Y.: A Neural-Network-Based Eye Pupil Reaction Model for Use with Television Programs, Japan Journal of Education Technology, Vol.18, No.2, pp.61–70 (1995) (in Japanese).
• [10]  Sejima, Y., Egawa, S., Maeda, R., Sato, S. and Watanabe, T.: A Speech-Driven Pupil Response Robot Synchronized with Burst-Pause of Utterance, Proceedings of the 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN2017), Lisbon, pp.437–442 (2017).
• [11]  Sejima, Y., Egawa, S., Sato, Y. and Watanabe, T.: A pupil response system using hemispherical displays for enhancing affective conveyance, Journal of Advanced Mechanical Design, Systems, and Manufacturing, Vol.13, No.2, pp.JAMDSM0032 (2019).
• [12]  Sejima, Y., Kawamoto, H., Sato, Y. and Watanabe, T.: Effects of pupil area on impression formation in pupil expression media Transactions of the JSME (in Japanese), Vol.87, No.903, pp.21-00187 (2021). (in Japanese).
• [13]  Honma, K.: Standard physiology, Igaku-Shoin Ltd. (2019), p.292 (in Japanese).
• [14]  Kret, M. E., Fischer, A. H. and De Dreu, C. K.: Pupil mimicry correlates with trust in in-group partners with dilating pupils, Psychological science, Vol.26, No.9, pp.1401–1410 (2015).
• [15]  Sejima, Y. and Watanabe, T.: Development of a speech-driven embodied listening system by mirroring of pupil response, Proceedings of the Symposium on Human Interface, Sendai, pp.560-/564 (2021) (in Japanese).
• [16]  Hess,E.H.: The Role of Pupil Size in Communication, Scientific American, Vol.233, No.5, pp.110–119 (1975).
• [17]  Watanabe, T., Okubo, M., Nakashige, M. and Danbara, R.: InterActor: Speech-Driven Embodied Interactive Actor, International Journal of Human-Computer Interaction, Vol.17, No.1, pp.43–60 (2004).
• [18]  Yoshida, M., Watanabe, T. and Yamamoto, M.: Development of a Speech-Driven Embodied Entrainment System with 3DCG Objects, Transactions of Human Interface Society, Vol.9, No.3, pp.87–96 (2007) (in Japanese).
• [19]  Sejima, Y., Watanabe, T. and Jindai, M.: Estimation Model of Interaction-activated Communication based on the Heat Conduction Equation, Journal of Advanced Mechanical Design, Systems, and Manufacturing, Vol.10, No.9, pp.JAMDSM0103 (2016).
• [20]  Sejima, Y. and Watanabe, T.: Evaluation of a speech-driven embodied listening system by mirroring of pupil response, Proceedings of the Symposium on Human Interface, Osaka, pp.110–116 (2022) (in Japanese).
• [21]  Takeuchi, K.: Mathematical Statistics in Phenomenon and Behavior, Shin-Yo-Sha Ltd. (1978), pp.133–148 (in Japanese).
• [22]  Sejima, Y., Sato, Y., Watanabe, T. and Jindai, M.: Speech-driven Embodied Entrainment Character System with Pupillary Response, Bulletin of the Mechanical Engineering Journal, Vol.3, No.4, pp.15-00314 (2016).
• [23]  Aoyagi, K.: An Experimental Study on Non-verbal Skills in a Situation of Clinical Psychology: Effects of Mirroring on Client’s Perception of Counselor Empathy, Japanese Journal of Counseling, Vol.46, No.2, pp.83–90 (2013).
• [24]  Hess, E. H.,: Attitude and Pupil Size, Scientific American, Vol.212, No.4, pp.46–54 (1965).
• [25]  Rogers, S. L., Guidetti, O., Speelman, C.P., Longmuir, M. and Phillips, R.: Contact Is in the Eye of the Beholder: The Eye Contact Illusion, Perception, Vol.48, No.3, pp.248–252 (2019).