Lossless Coding for Individual Sequences

Abstract

This paper deals with the coding problem of individual sequences, focusing on both fixed and variable-length coding methods. In the fixed-length coding method, we show that if coding errors are not allowed, the minimum coding rate coincides with some information measures, such as topological entropy, and we also explain a concrete coding method. If the error rate vanishes as the block length tends to infinity, we show that the minimum coding rate is characterized by information measures, such as Ziv entropy, and we also explain an efficient coding method. We then show that the variable-length minimum coding rate coincides with the entropy of the empirical distribution of individual sequences, and that this rate can be achieved by using the LZ78 code. Further, we discuss a sufficient condition on the universal coding for a set of individual sequences on a countably infinite alphabet. Finally, we introduce the concept of essential optimality for the encoding of finite length sequences and show that the LZ77 code and the universal context tree coding are essentially optimal.