Investigation on influence of speech bubbles augmented to speakers in videos on listeners’ attitude interpretation

Abstract

Online conversations through platforms like Zoom or AR-enabled dialogues with smart glasses offer the unique advantage of overlaying relevant information on screens. Recent research has introduced systems that augment speakers with speech bubbles containing either identical or alternative text to their verbal statements, aiming to enhance dialogue. However, these systems present an interpretive challenge: when spoken content and text in speech bubble convey different attitudes, it remains unclear which source listeners prioritize to form impressions of speakers. This study investigates how speech bubble augmentation influences listeners’ interpretation of speaker attitudes. We categorized these influences into three types: clarification, amplification/attenuation, and transformation of attitudes. Through three online experiments designed to examine each category, we confirmed all three types of influence. Notably, listeners demonstrated a tendency to prioritize speech bubble content over verbal statements when interpreting speaker attitudes. Our findings provide fundamental insights into the effects of speech bubble augmentation on human communication in digital interaction environments while acknowledging limitations in experimental scope and demographics that necessitate further research across diverse contexts.