Generating Referring Expressions based-on Perceptual Grouping

Abstract

Past work of generating referring expressions mainly utilized attributes of objects and binary relations between objects to distinguish the referent from other objects. However, such an approach does not work well when there is no distinctive attribute among objects.To overcome this limitation, this paper proposes a novel generation method utilizing the perceptual groups of objects and n-ary relations among them. With the proposed method, an expression like “the rightmost ball in the left cluster of three balls” can be generated.The key is to identify groups of objects that are naturally recognized by humans.We conducted psychological experiments with 42 subjects to collect referring expressions in such situations, and built a generation algorithm based on the results.The evaluation using another 23 subjects showed that the proposed method could effectively generate proper referring expressions.