Annotator Attribute Fair Label Aggregation

Abstract

Supervised learning and model evaluation in machine learning domain require labeled data, and the labeling is mainly done manually. However, due to the large variability in human annotation quality, labels are redundantly collected and aggregated. With growing interest in the fairness of predictive models in machine learning, we focused on the fairness of label aggregation algorithms used to construct supervised data. In this study, we propose unbiased label aggregation algorithms for annotator attributes. We propose three fairness methods: data separation, sample weighting, and E-step correction, and combine them with existing label aggregation algorithms. We compare the results of the label aggregation algorithm combined with the proposed method using the U.S. recidivism prediction annotation dataset. The experimental results show that the proposed methods increase the influence of labels by minority annotators and achieves fair label aggregation with a trade-off of a slight decrease in accuracy.