Abstract
This paper argues about consistency of syntactic dependency corpus annotation. Dependency structure involves unambiguous substructures specified by syntactic constraints and underspecified substructures evaluated by selectional preferences of annotators. In most cases, the syntactic constraints have a priority over the selectional preferences. In some cases, however, the selectional preferences are given priority over the syntactic constraints. We propose a new psycholinguistic experimental environment to use a fitness game application, in which a user stands on Nintendo Wii Balance Board and plays a parsing game competing on the accuracy and the speed. Tha game-based experiments investigate how correctly a human can parse written sentences. We evaluate syntactically confusing sentences with 13 experimental subjects and 6 dependency parsers. Moreover, we investigate effects of selectional preferences in the incorrectly annotated sentences.
