Linking Pronouns in Dialogue with Whiteboard Symbols: Towards Confusion Detection in Math Collaboration

Abstract

In collaborative problem-solving, particularly in technical domains like mathematics, discussions often combine spoken dialogue with a shared visual space, such as a whiteboard. A critical challenge for comprehending these interactions is resolving the reference between ambiguous expressions in dialogue (e.g., pronouns) and the specific symbols or equations written on the board.To address this, and drawing inspiration from research in Visually-Grounded Dialogue, we propose a new annotation schema for capturing the discourse structure of these multimodal discussions by explicitly linking dialogue utterances to their corresponding element on the whiteboard.