An adaptive test based on a two-alternative forced choice task for benchmarking the reading abilities of humans and AI in image diagnosis

Abstract

To enable effective collaboration between healthcare professionals and image diagnosis systems in medical settings, it is necessary to comprehensively compare the reading abilities of individual healthcare workers and the diagnosis systems used in these settings. In this study, we developed an adaptive test for medical image diagnosis abilities that can be used in common by both humans and image diagnosis systems. The test requires participants to make a binary judgment of positive or negative when viewing an image composed of multidimensional features. We designed an acquisition function that determines the search point for the next trial with the steepest slope of the stimulus characteristic curve based on the responses. Although this study provided a framework for comparing the reading abilities of humans and AI, further improvements are needed in the setting of the acquisition function.