Renyi entropy

Following the discussion we had here about the merits of Tsallis and Renyi entropies, here's an interesting paper by Peter Harremöes - Interpretations of Renyi Entropies And Divergences. Harremöes is looking for an information theoretic interpretation of the Renyi entropies in terms of what he calls an operational definition:

To us an operational definition of a quantity means that the quantity is the natural way to answer a natural question and that the quantity can be estimated by feasible measurements combined with a reasonable number of computations. In this sense the Shannon entropy has an operational definition as a compression rate and the Kolmogorov entropy has an operational definition as shortest program describing data. (p. 2)

Via an introductory account of codes, we learn that "the Renyi divergence measures how much a probabilistic mixture of two codes can be compressed".

Like the Kullback-Leibler divergence (Shannon relative entropy), the Renyi divergence is addititive or extensive in the sense that

D_q(P₁ x P₂//Q₁ x Q₂) = D_q(P₁//Q₁) + D_q(P₂//Q₂).

[KL-divergence equals the Renyi divergence for q = 1.] So too is the corresponding Renyi entropy. But for q > 1 it lacks a property possessed by the Shannon entropy, and also by all Renyi entropies with q in [0,1], namely concavity. The Tsallis entropy chooses the other option, and so while concave for q > 1, it is no longer additive/extensive.