A Method for Inspecting Neural NLP Models Based on Text That Activates Neurons

Abstract

In this study, as an approach to provide a deeper insight into neural Natural Language Processing (NLP) models, we propose a method for analyzing the roles of individual neurons, which is the finest component of the models, by observing sentences which strongly activate individual neurons. Our method retrieves the sentences from the massive corpora, and abstracts the sentences for interpretation using data-mining techniques. In the experiments, we demonstrate that our method can give thoughtful insights into what linguistic aspects each neuron of a given model captures as well as how multiple neurons relate to each other.