Natural distributions

I've been thinking of late about what we can say about the probability distributions we might expect the world to throw at us. This is very important in machine learning. An algorithm which, knowingly or not, 'expected' to meet with certain specific kinds of distribution which in fact did occur is clearly at an advantage. But what kinds of consideration are important here? The universality of the Gaussian normal distribution as made explicit in the Central Limit Theorem might provide a pointer. Are there other distributions which arise robustly in different situations?

In Universality for mathematical and physical systems, Percy Deift describes distributions which govern data as far removed as the spacings between parked cars and the (scaled) spacings between the zeros of the Riemann zeta function. These distributions are studied in a field known as random matrix theory, which considers, for example, the eigenvalues of a random orthogonal matrix. Answers are beginning to emerge as to why these distributions are encountered, which parallel the three components of the central limit theorem: a statistical component (take independent, identically distributed random variables, centered and scaled), an algebraic component (add the variables), and an analytic component (take the limit in distribution as n → infinity). He signs off with the intriguing comment:

Our final comment/speculation is on the space D, say, of probability distributions. A priori, D is just a set without any "topography". But we know at least one interesting point on D, the Gaussian distribution F_G. By the central limit theorem, it lies in a "valley", and nearby distributions are drawn towards it. What we seem to be learning is that there are other interesting distributions, like F₁ or F₂, etc., which also lie in "valleys" and draw nearby distributions in towards them. This suggests that we equip D with some natural topological and Riemannian structure, and study the properties of D as a manifold per se.