Analyzing Difficulty of Machine Comprehension focusing on Examples Answerable by BERT without Question

Abstract

In the field of machine comprehension (MC), the task of an MC model is to predict the answer (A) from a question (Q) and related context (C) of the question. However, in this paper, it is discovered that there exist examples that can be correctly answered by an MC model BERT where only the context of the example is given without the question being given, which means that the difficulty of examples of machine comprehension vary. Based on this finding, this paper proposes a method based on BERT which splits the training examples of the MC dataset SQuAD1.1 into “easy to answer” and “hard to answer” ones. Experimental evaluation results of comparing the two models, one of which is trained with the “easy to answer” examples only, while the other of which is trained with the “hard to answer” examples only, show that the latter outperforms the former.