Generation of zero-shot label sets and attribute classification of listening test dialogues

Abstract

Zero-shot classification will produce different classification results for the same texts depending on the input label set.In this paper, we propose a method to generate a large number of candidate label sets for the same zero-shot classification target by antonym substitution and conversion to synonyms using WordNet and find appropriate labels from themFour zero-shot classification methods are evaluated: 1. cosine similarity of text by BERT model, 2. cosine similarity of text by OpenAI's model, and 3. pre-trained zero-shot model of MoritzLauer.In the evaluation experiment, we collected 50 listening test dialogues from each of the N5 to N1 levels of the past Japanese Language Proficiency Test (JLPT) and classified them manually.Three classification attributes of Dialogue Location (6 categories), Speaker's Relationship (2 categories and 4 categories), and Dialogue Style (2 categories) were evaluated.We prepared 212 candidate label sets and counted the RMSE (Root Mean Square Error) of these labels for the four zero-shot classification methods. The results confirmed that the proposed method can obtain higher accuracy label sets for zero-shot classification.