A Crowdsourcing Approach to Integrate the Preferences for AI Metrics That is Different Based on Stakeholders

Abstract

Recently, fairness-aware artificial intelligence (AI) technology has been developed to remove discriminatory bias, e.g., bias on race and gender. On the other hand, there are trade-offs among the metrics of accuracy and fairness in AI models and different stakeholders have different preferences for the metrics. Hence, to form an agreement on the preferences, existing research has explored workshop approaches encouraging dialogue among stakeholders. However, it is practically difficult for multiple stakeholders to have conversations at the same place and time. In this paper, we propose a method to aggregate the preference of each stakeholder using an online survey. In our study, 739 crowdsourced participants are randomly divided into 4 stakeholder groups and asked to rank 5 machine learning models. Through this survey, we calculate the preference of each stakeholder group. We examine the preference of each stakeholder and whether the information on the other stakeholder affects a stakeholder's preference for the metrics. As a result, through our method, the preference of each stakeholder successfully meets the requirements of their role. On the other hand, it is clarified that the preference for the metrics is not affected by the information on the other stakeholders.