Information Geometry (Part 2)

October 23, 2010

Information Geometry (Part 2)

John Baez

Last time I provided some background to this paper:

Gavin E. Crooks, Measuring thermodynamic length.

Now I'll tell you a bit about what it actually says!

Remember the story so far: we've got a physical system that's in a state of maximum entropy. I didn't emphasize this yet, but that happens whenever our system is in thermodynamic equilibrium. An example would be a box of gas inside a piston. Suppose you choose any number for the energy of the gas and any number for its volume. Then there's a unique state of the gas that maximizes its entropy, given the constraint that on average, its energy and volume have the values you've chosen. And this describes what the gas will be like in equilibrium!

Remember, by 'state' I mean mixed state: it's a probabilistic description. And I say the energy and volume have chosen values on average because there will be random fluctuations. Indeed, if you look carefully at the head of the piston, you'll see it quivering: the volume of the gas only equals the volume you've specified on average. Same for the energy.

More generally: imagine picking any list of numbers, and finding the maximum entropy state where some chosen observables have these numbers as their average values. Then there will be fluctuations in the values of these observables — thermal fluctuations, but also possibly quantum fluctuations. So, you'll get a probability distribution on the space of possible values of your chosen observables. You should visualize this probability distribution as a little fuzzy cloud centered at the average value!

To a first approximation, this cloud will be shaped like a little ellipsoid. And if you can pick the average value of your observables to be whatever you'll like, you'll get lots of little ellipsoids this way, one centered at each point. And the cool idea is to imagine the space of possible values of your observables as having a weirdly warped geometry, such that relative to this geometry, these ellipsoids are actually spheres.

This weirdly warped geometry is an example of an 'information geometry': a geometry that's defined using the concept of information. This shouldn't be surprising: after all, we're talking about maximum entropy, and entropy is related to information. But I want to gradually make this idea more precise. Bring on the math!

We've got a bunch of observables $X_{1}, \dots, X_{n}$ , and we're assuming that for any list of numbers $x_{1}, \dots, x_{n}$ , the system has a unique maximal-entropy state $ρ$ for which the expected value of the observable $X_{i}$ is $x_{i}$ :

⟨ X_{i} ⟩ = x_{i}

This state $ρ$ is called the Gibbs state and I told you how to find it when it exists. In fact it may not exist for every list of numbers $x_{1}, \dots, x_{n}$ , but we'll be perfectly happy if it does for all choices of

x = (x_{1}, \dots, x_{n})

lying in some open subset of $R^{n}$

By the way, I should really call this Gibbs state $ρ (x)$ or something to indicate how it depends on $x$ , but I won't usually do that. I expect some intelligence on your part!

Now at each point $x$ we can define a covariance matrix

⟨ (X_{i} - ⟨ X_{i} ⟩) (X_{j} - ⟨ X_{j} ⟩) ⟩

If we take its real part, we get a symmetric matrix:

g_{i j} = Re ⟨ (X_{i} - ⟨ X_{i} ⟩) (X_{j} - ⟨ X_{j} ⟩) ⟩

It's also nonnegative — that's easy to see, since the variance of a probability distribution can't be negative. When we're lucky this matrix will be positive definite. When we're even luckier, it will depend smoothly on $x$ . In this case, $g_{i j}$ will define a Riemannian metric on our open set.

So far this is all review of last time. Sorry: I seem to have reached the age where I can't say anything interesting without warming up for about 15 minutes first. It's like when my mom tells me about an exciting event that happened to her: she starts by saying "Well, I woke up, and it was cloudy out..."

But now I want to give you an explicit formula for the metric $g_{i j}$ , and then rewrite it in a way that'll work even when the state $ρ$ is not a maximal-entropy state. And this formula will then be the general definition of the 'Fisher information metric' (if we're doing classical mechanics), or a quantum version thereof (if we're doing quantum mechanics).

Crooks does the classical case — so let's do the quantum case, okay? Last time I claimed that in the quantum case, our maximum-entropy state is the Gibbs state

ρ = \frac{1}{Z} e^{- λ^{i} X_{i}}

where $λ^{i}$ are the 'conjugate variables' of the observables $X_{i}$ , we're using the Einstein summation convention to sum over repeated indices that show up once upstairs and once downstairs, and $Z$ is the partition function

Z = tr (e^{- λ^{i} X_{i}})

(To be honest: last time I wrote the indices on the conjugate variables $λ^{i}$ as subscripts rather than superscripts, because I didn't want some poor schlep out there to think that $λ^{1}, \dots, λ^{n}$ were the powers of some number $λ$ . But now I'm assuming you're all grown up and ready to juggle indices! We're doing Riemannian geometry, after all.)

Also last time I claimed that it's tremendously fun and enlightening to take the derivative of the logarithm of $Z$ . The reason is that it gives you the mean values of your observables:

⟨ X_{i} ⟩ = - \frac{\partial}{\partial λ^{i}} \ln Z

But now let's take the derivative of the logarithm of $ρ$ . Remember, $ρ$ is an operator — in fact a density matrix. But we can take its logarithm as explained last time, and the usual rules apply, so starting from

ρ = \frac{1}{Z} e^{- λ^{i} X_{i}}

we get

\ln ρ = - λ^{i} X_{i} - \ln Z

Next, let's differentiate both sides with respect to $λ^{i}$ . Why? Well, from what I just said, you should be itching to differentiate $\ln Z$ . So let's give in to that temptation:

\frac{\partial}{\partial λ^{i}} \ln ρ = - X_{i} + ⟨ X_{i} ⟩

Hey! Now we've got a formula for the 'fluctuation' of the observable $X_{i}$ — that is, how much it differs from its mean value:

X_{i} - ⟨ X_{i} ⟩ = - \frac{\partial \ln ρ}{\partial λ^{i}}

This is incredibly cool! I should have learned this formula decades ago, but somehow I just bumped into it now. I knew of course that $\ln ρ$ shows up in the formula for the entropy:

S (ρ) = tr (ρ \ln ρ)

But I never had the brains to think about $\ln ρ$ all by itself. So I'm really excited to discover that it's an interesting entity in its own right — and fun to differentiate, just like $\ln Z$ .

Now we get our cool formula for $g_{i j}$ . Remember, it's defined by

g_{i j} = Re ⟨ (X_{i} - ⟨ X_{i} ⟩) (X_{j} - ⟨ X_{j} ⟩) ⟩

But now that we know

X_{i} - ⟨ X_{i} ⟩ = - \frac{\partial \ln ρ}{\partial λ^{i}}

we get the formula we were looking for:

g_{i j} = Re ⟨ \frac{\partial \ln ρ}{\partial λ^{i}} \frac{\partial \ln ρ}{\partial λ^{j}} ⟩

Beautiful, eh? And of course the expected value of any observable $A$ in the state $ρ$ is

⟨ A ⟩ = tr (ρ A)

so we can also write the covariance matrix like this:

g_{i j} = Re tr (ρ \frac{\partial \ln ρ}{\partial λ^{i}} \frac{\partial \ln ρ}{\partial λ^{j}})

Lo and behold! This formula makes sense whenever $ρ$ is any density matrix depending smoothly on some parameters $λ^{i}$ . We don't need it to be a Gibbs state! So, we can work more generally.

Indeed, whenever we have any smooth function from a manifold to the space of density matrices for some Hilbert space, we can define $g_{i j}$ by the above formula! And when it's positive definite, we get a Riemannian metric on our manifold: the Bures information metric.

The classical analogue is the somewhat more well-known 'Fisher information metric'. When we go from quantum to classical, operators become functions and traces become integrals. There's nothing complex anymore, so taking the real part becomes unnecessary. So the Fisher information metric looks like this:

g_{i j} = \int_{Ω} p (ω) \frac{\partial \ln p (ω)}{\partial λ^{i}} \frac{\partial \ln p (ω)}{\partial λ^{j}} d ω

Here I'm assuming we've got a smooth function $p$ from some manifold $M$ to the space of probability distributions on some measure space $(Ω, d ω)$ . Working in local coordinates $λ^{i}$ on our manifold $M$ , the above formula defines a Riemannian metric on $M$ , at least when $g_{i j}$ is positive definite. And that's the Fisher information metric!

Crooks says more: he describes an experiment that would let you measure the length of a path with respect to the Fisher information metric — at least in the case where the state $ρ (x)$ is the Gibbs state with $⟨ X_{i} ⟩ = x_{i}$ . And that explains why he calls it 'thermodynamic length'.

There's a lot more to say about this, and also about another question: What use is the Fisher information metric in the general case where the states $ρ (x)$ aren't Gibbs states?

But it's dinnertime, so I'll stop here.

You can read a discussion of this article on Azimuth, and make your own comments or ask questions there!