Automatic Generation of Datasets with High Evaluation Capability for Machine Learning Algorithms Using Item Response Theory

Abstract

We propose a framework for generating datasets that can appropriately evaluate the performance of the proposed algorithms in competitions. In most competitions which students and engineers studying machine learning participate in, the dataset is selected ad hoc. Therefore, there has been an issue, such as the use of dataset that would yield high performance no matter what algorithm is used. To resolve these problems, we conbine Item Response Theory and Conditional VAE. Item Response Theory is a theory for creating test questions and evaluating examinees' abilities. Conditional VAE is a method for generating images according to parameters. Experimental results show that our method generates a dataset which can evaluate the performance of algorithms appropriately more than MNIST.