Waiting one second boosted data quality: A simple and general-purpose solution to allocate the trade-off between annotators' performance and workload

Abstract

The development of statistical learning techniques generally requires large, accurately annotated data sets. However, for tasks where the definition of the correct label cannot be uniquely defined, especially when the task is highly specialized such as medical data, it is difficult to obtain large, accurately annotated data sets. We hypothesized that there exists an appropriate thinking time that balances the trade-off between accuracy and mental strain. We tested the effect of an intervention in which participants were prevented from answering for a certain period of time after the image was presented to them when deciding whether a medical image was abnormal or normal. In two behavioral experiments (physicians (N=634)), the expectation of a correct response increased when the image was made unanswerable for one second after presentation. This study showed that annotation quality can be improved in a simple and cost-effective way by utilizing human cognitive characteristics.