Monotonicity Dataset Creation on Crowdsourcing

Abstract

Large crowdsourced datasets are widely used for training and evaluating neural models on recognizing textual entailment (RTE). However, it is still unclear whether neural models can capture logical inferences, including monotonicity reasoning, for which no large naturalistic dataset has yet been developed. To investigate this issue, we introduce a method of creating a dataset for monotonicity reasoning by crowdsourcing and report the result of the first run. The error analysis indicates that workers tend to provide different answers from what logical entailment defines, for some downward monotonicity reasonings involving pragmatic reasoning.