Self-Ensemble: Integration of N-best Hypotheses by Lexically Constrained Decoding

Abstract

In language generation, a reranking method improves the quality of generated sentences by re-scoring the top N hypotheses. The reranking method assumes the existence of higher quality hypotheses in N-best. We expand this assumption to be more practical as partly higher quality hypotheses exist in the N-best; however, they may be imperfect as the entire sentence. We propose a method for generating high-quality outputs by integrating high-quality fragments in the N-best. Specifically, we first obtain the N-best hypotheses and estimate the quality of each token. We then perform decoding again applying lexical constraints, with the words predicted to be wrong as negative constraints and those predicted to be correct as positive constraints. This method produces sentences that contain the correct words included in the N-best output and do not contain the wrong words. Empirical experiments on paraphrase generation, summarization, translation, and constrained text generation confirmed that our method outperformed strong N-best reranking methods in paraphrase generation and summarization tasks.