An Extension of Dawid-Skene Model Considering Diversity of Labels

Abstract

Crowdsourcing is a popular way to obtain labeled data at relatively low cost. Existing methods for inferring true labels from crowdsourced noisy annotations assume that there is one true label for each item, but sometimes it is more appropriate to assume that each item has multiple true labels depending on the attributes of the workers. In this study, we propose a model that estimates the abilities and labels based on worker attributes, which considers a diversity of labels. The proposed model is an extension of the Dawid-Skene model and assumes that there is a true label for each combination of attributes. Experiments on synthesis data, where each item had multiple true labels depending on the attributes of the workers, showed that the existing model underestimated the ability of a minority group, but the proposed method accurately estimated it.