Analyzing Transformers via Value Matrices

Abstract

We propose a new method to analyze Transformer language models. In Transformer self-attention modules, attention weights are calculated from the query vectors and key vectors. Then, output vectors are obtained by taking the weighted sum of value vectors. While existing works on analysis of Transformer have focused on attention weights, this work focused on value and output matrices. We obtain joint matrices by multiplying both matrices, and show that the trace of the joint matrices are correlated with word co-occurences.