A Simple and Effective Usage of Word Clusters for CBOW Model

Abstract

We propose a simple and effective method for incorporating word clusters into the Continuous Bag-of-Words (CBOW) model. Specifically, we propose replacing infrequent input and output words in CBOW with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of frequent words and a small amount of cluster embeddings, which will be fine-tuned in downstream tasks. We empirically demonstrate that our replacing method works well on several downstream tasks. Through our analysis, we also show that our method is potentially useful for other similar models that produce word embeddings.