Huffman coding is the most well known code tree problem, but there are a number of interesting variations of the problem which are less well known, and there are many results available concerning the properties of Huffman codes. This paper addresses the Huffman code tree problem and characteristics of the Huff man code tree along with several problem variations, specifically, when the source alphabet is infinite, when a lexicographic constrain is imposed on the codewords, or, that is, the Hu-Tucker code tree problem, when a maximum length constraint is imposed on the codewords, and when other tree characteristics besides average codeword length are minimized.

View full abstract

The redundancy of a universal data compression code is measured by the Kullback-Leibler distance between the unknown source and the model of the code. MDL (Minimum Description Length) code, introduced by J. Rissanen, achieves the redundancy rate of (k/2n) log n in encoding a sequence of lengh n from the source in a class of distributions described by k-dimensional parameters. It is sharpend by the studies on Bayes code.

View full abstract

This paper surveys some universal codes for text compression from a unifying viewpoint. As is well known in information theory, the data compression limit of a source is given by its entropy rate. Since universal codes are designed to work with an arbitrary source distribution, they can be expected to serve as entropy estimators. In order to reveal the capability of estimating the entropy, we introduce the notion of a context table, in which all the substrings in a text are enumerated in lexicographic order. A context table is useful to intuitively understand several important quantities including the recurrence time of a substring, the conditional recurrence time, and the length of the shortest unique substring. In terms of these quantities we characterize the Ziv-Lempel code and sort-based symbol ranking codes. Some relations among these apparently independent codes are established by the help of context table concepts.

View full abstract

We discuss the following problems of random number generation. Problem 1: How many fair coin flips can we generate from a discrete random variable X with distribution p=(p_1, p_2, …, P_M)? Or, generation of unbiased N-sided coin from an information source X (intrinsic randomness problem). Problem 2: How to generate a discrete random variable Y∈{1, 2, …, N} with probalility q=(q_1, q_2, …, q_N) by using a fair coin. Or, generation of random number Y by using an unbiased M-sided coin (resolvability problem). How many fair coin flips does it take to generate a random variable X? Problem 3: How to generate a discrete random variable Y∈{1, 2, …, N} with probability q=(q_1, q_2, …, q_N) from a discrete random variable X with probability p=(p_1, p_2, …, P_M). Lastly, we discuss the relations between the problems above and source coding problems.

View full abstract

In most cases treated in information theory, stationary memoryless/ergodic property on the system are assumed. In this paper, source coding theorems on the general information sources, which means neither stationary memoryless nor stationary ergodic property are conditioned, are considered. We clarify the following two issues: 1) What quantity can be essential in source coding problems under the general sources? 2) What are the differences between theorems under conventional stationary memoryless/ergodic condition and ones under the general sources?

View full abstract