Information Geometry

John Baez

October 20, 2010

Information geometry is the study of 'stochastic manifolds', which are spaces where each point is a hypothesis about some state of affairs. This subject, usually considered a branch of statistics, has important applications to machine learning and somewhat unexpected connections to evolutionary biology. To learn this subject, I'm writing a series of articles on it. You can navigate forwards and back through these using the blue arrows. And by clicking the links that say "on Azimuth", you can see blog entries containing these articles. Those let you read comments about my articles—and also make comments or ask questions of your own!

Part 1 - the Fisher information metric from statistical mechanics.
Part 2 - connecting the statistical mechanics approach to the usual definition of the Fisher information metric.
Part 3 - the Fisher information metric on any manifold equipped with a map to the mixed states of some system.
Part 4 - the Fisher information metric as the real part of a complex-valued quantity whose imaginary part measures quantum uncertainty.
Part 5 - an example: the harmonic oscillator in a heat bath.
Part 6 - relative entropy.
Part 7 - the Fisher information metric as the matrix of second derivatives of relative entropy.
Part 8 - information geometry and population biology: how natural selection resembles Bayesian inference, and how it's related to relative entropy.

You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage. - John von Neumann, giving advice to Claude Shannon on what to name his discovery.