Crowdsourcing Worker Development based on Probabilistic Task Network

Abstract

Current crowdsourcing platforms such as Amazon Mechanical Turk provide an attractive solution Crowdsourcing platforms provide an attractive solution for processing numerous tasks at a low cost. However, insufficient quality control remains a major concern. Therefore, we developed a private crowdsourcing system that allows us to devise quality control methods. In the present study, we propose a grade-based training method for workers in order to avoid simple exclusion of low-quality workers and shrinkage of the crowdsourcing market in the near future. Our training method utilizes probabilistic networks to estimate correlations between tasks based on workers’ records for 18.5 million tasks and then allocates pre-learning tasks to the workers to raise the accuracy of target tasks according to the task correlations. In an experiment, the method automatically allocated 31 pre-learning task categories for 9 target task categories, and after the training of the pre-learning tasks, we confirmed that the accuracy of the target tasks was raised by 7.8 points on average. This result was comparatively higher than those of pre-learning tasks allocated using other methods, such as decision trees. We thus confirmed that the task correlations can be estimated using a large amount of worker records, and that these are useful for the grade-based training of low-quality workers.