Information geometry

Information geometry is a branch of mathematics that applies the techniques of differential geometry to the field of probability theory. This is done by taking probability distributions for a statistical model as the points of a Riemannian manifold, forming a statistical manifold. The Fisher information metric provides the Riemannian metric.

Information geometry reached maturity through the work of Shun'ichi Amari and other Japanese mathematicians in the 1980s. Amari and Nagaoka's book, Methods of Information Geometry,[1] is cited by most works of the relatively young field due to its broad coverage of significant developments attained using the methods of information geometry up to the year 2000. Many of these developments were previously only available in Japanese-language publications.

Introduction

The following introduction is based on Methods of Information Geometry.[1]

Information and probability

Define an n-set to be a set V with cardinality |V|=n. To choose an element v (value, state, point, outcome) from an n-set V, one needs to specify \log_bn b-sets (default b=2), if one disregards all but the cardinality. That is, I(v)=\log n nats of information are required to specify v; equivalently, I(v)=\log_2 n bits are needed.

By considering the occurrences C of values from V, one has an alternate way to refer to v\in V, through C. First, one chooses an occurrence c\in C, which requires information of I(c)=\log_2 |C| bits. To specify v, one subtracts the excess information used to choose one c from all those linked to v, this is I(c_v)=\log_2 |C_v|. Then, \frac{|C|}{|C_v|} is the number of |C_v| portions fitting into |C|. Thus, one needs I(v)=\log_2\frac{|C|}{|C_v|} bits to choose one of them. So the information (variable size, code length, number of bits) needed to refer to v, considering its occurrences in a message is


I(v)=-\log_2 p(v)

Finally, p(v)I(v) is the normalized portion of information needed to code all occurrences of one v. The averaged code length over all values is H(V)=-\sum p(v)\log p(v). H(V) is called the entropy of a random variable V.

Statistical model, Parameters

With a probability distribution p one looks at a variable V through an observation context like a message or an experimental setup.

The context can often be identified by a set of parameters through combinatorial reasoning. The parameters can have an arbitrary number of dimensions and can be very local or less so, as long as the context given by a certain \xi=[\xi^i]\in\mathbb{R}^n produces every value of V, i.e. the support \text{supp}(V) does not change as function of \xi. Every \xi determines one probability distribution for V. Basically all distributions for which there exists an explicit analytical formula fall into this category (Binomial, Normal, Poisson, ...). The parameters in these cases have a concrete meaning in the underlying setup, which is a statistical model for the context of V.

The parameters are quite different in nature from V itself, because they do not describe V, but the observation context for V.

A parameterization of the form

p(v)=\sum\xi^i p_i(v)=\xi^i p_i

with

\sum p_i(v_j)=1 and \sum\xi^i=1,

that mixes different distributions p_i(v), is called a mixture distribution, mixture or m-parameterization or mixture for short. All such parameterizations are related through an affine transformation \rho=A\xi+B. A parameterization with such a transformation rule is called flat.

A flat parameterization for I(v)=\log p(v)=E(v)+\sum\xi^iF_i(v) is an exponential or e parameterization, because the parameters are in the exponent of p(v). There are several important distributions, like Normal and Poisson, that fall into this category. These distributions are collectively referred to as exponential family or e-family. The p-manifold for such distributions is not affine, but the \log p manifold is. This is called e-affine. The parameterization \log p(v)=E(v)+\sum\xi^iF_i(v)-\psi(\xi) for the exponential family can be mapped to the one above by making \psi(\xi) another parameter and extend [F_i]\rightarrow[F_i,1].

Differential geometry applied to probability

In information geometry, the methods of differential geometry are applied to describe the space of probability distributions for one variable V. This is done by using a coordinate or atlas \xi\in\mathbb{R}^n. Furthermore, the probability p(v;\xi) must be a differentiable and invertible function of \xi. In this case, the [\xi^i] are coordinates of the p(v;\xi)-space, and the latter is a differential manifold M.

Derivatives are defined as is usual for a differentiable manifold:

\partial_i f = \frac{\partial f}{\partial\xi^i}:=\frac{\partial\bar{f}}{\partial\xi^i}

with \bar{f}=f\circ \xi^{-1}, for f\in\mathcal{F}(M) a real-valued function on M.

Given a function f on M, one may "geometrize" it by taking it to define a new manifold. This is done by defining coordinate functions on this new manifold as

\phi=(f\circ\xi^{-1})^{-1}=\xi\circ f^{-1}.

In this way one "geometricizes" a function f, by encoding it into the coordinates used to describe the system.

For f=\log the inverse is f^{-1}=\exp and the resulting manifold of \log p points is called the e-representation. The p manifold itself is called the m-representation. The e- or m-representations, in the sense used here, does not refer to the parameterization families of the distribution.

Tangent space

Main article: Tangent space

In standard differential geometry, the tangent space on a manifold M at a point q is given by:


T_qM=\left\{X^i\partial_i\Big|X\in \mathbb{R}^n, \partial_i=\frac{\partial}{\partial \xi^i}\right\}

In ordinary differential geometry, there is no canonical coordinate system on the manifold; thus, typically, all discussion must be with regard to an atlas, that is, with regard to functions on the manifold. As a result, tangent spaces and vectors are defined as operators acting on this space of functions. So, for example, in ordinary differential geometry, the basis vectors of the tangent space are the operators \partial_i.

However, with probability distributions p(v;\xi), one can calculate value-wise. So it is possible to express a tangent space vector directly as X^i\partial_ip ( m-representation ) or X^i\partial_i\log p ( e-representation ), and not as operators.

alpha representation

Important functions f of p are coded by a parameter \alpha with the important values 1, 0 and -1:

Distributions that allow a flat parameterization \ell^{\alpha}(v;\xi)=E(v)+\xi^iF_i(v) are called collectively \alpha-family ( m-, e- or 0-family ) of distributions and the according manifold is called \alpha-affine.

The \alpha tangent vector is X^{(\alpha)}=X^i\partial_i\ell^{\alpha}.

Inner product

One may introduce an inner product on the tangent space of manifold M at point q as a linear, symmetric and positive definite map

\langle\;,\;\rangle_q:T_q\times T_q\to\mathbb{R}.

This allows a Riemannian metric to be defined; the resulting manifold is a Riemannian manifold. All of the usual concepts of ordinary differential geometry carry over, including the norm

||X||=\sqrt{\langle X,X\rangle},

the line element ds, the volume element dV, and the cotangent space

T_q^*M=\{T_q\rightarrow\mathbb{R}\}

that is, the dual space to the tangent space T_q. From these, one may construct tensors, as usual.

Fisher metric as inner product

For probability manifolds such an inner product is given by the Fisher information metric.

Here are equivalent formulas of the Fisher information metric.

This relation with a divergence D(p||q) will be revisited further down.

The Fisher metric is motivated by

Affine connection

Main article: Affine connection

Like commonly done on Riemann manifolds, one may define an affine connection (or covariant derivative)

\nabla:TM\times TM\rightarrow TM

Given vector fields X and Y lying in the tangent bundle TM, the affine connection \nabla_XY describes how to differentiate the vector field Y along the directionX. It is itself a vector field; it is the sum of the infinitesimal change in the vector field Y, as one moves along the direction X, plus the infinitesimal change of the vector Y due to its parallel transport along the direction X. That is, it takes into account the changing nature of what it means to move a coordinate system in a "parallel" fashion, as one moves about in the manifold. In terms of the basis vectors \partial_k, one has the components:

\left(\nabla_XY\right)^k=X^i\left(\nabla_iY\right)^k=X^i(\partial_iY^k+Y^j\Gamma_{ij}^k)

The \Gamma_{ij}^k are Christoffel symbols. The affine connection may be used for defining curvature and torsion, like is usual in Riemannian geometry.

Alpha connection

A non-metric connection is not determined by a metric tensor g_{ij}; instead, it is and restricted by the requirement that the parallel transport \Pi_{q,{q'}}(\partial_i) between points q and q' must be a linear combination of the base vectors in T_{q'}M. Here,

\Pi_{q,q'}(\partial_j)=(\partial_j)_{q'}-d\xi^i\Gamma_{ij}^k(\partial_k)_{q'}

expresses the parallel transport of \partial_j as linear combination of the base vectors in T_{q'}M, i.e. the new \partial_j minus the change. Note that it is not a tensor (does not transform as a tensor).

For such a metric, one can construct a dual connection \nabla^* to make

\partial_ig_{jk}=\langle\nabla_{\partial_i}\partial_j,\partial_k\rangle+\langle\partial_j,\nabla^*_{\partial_i}\partial_k\rangle=\Gamma_{ij,k}+\Gamma_{ik,j}*=0,

for parallel transport using \nabla and \nabla^*.

For the mentioned \alpha-families the affine connection is called the \alpha-connection and can also be expressed in more ways.

For \alpha=\pm 1,0:

i.e. \Gamma_{ij,k}^{(-\alpha)} is dual to \Gamma_{ij,k}^{(\alpha)} with respect to the Fisher metric.
\Gamma_{ij,k}^{(-1)}=\sum\partial_i\partial_j(\xi^ip_i)\partial_k\ell^{(1)}=0,
i.e. 0-affine, and hence \Gamma_{ij,k}^{(1)}=0, i.e. 1-affine.

Divergence

A function of two distributions (points) D(p||q)\geq 0 with minimum 0 for p=q entails D[\partial_i||]=\partial_iD(p||p)=0 and D[||\partial_j]=\partial'_jD(p||p)=0. \partial_i is applied only to the first parameter, and \partial'_i only to the second. \partial_i is the direction, which brought the two points to be equal, when applied to the first parameter, and to diverge again, when applied to the second parameter, i.e. D[\partial_i||]=-D[||\partial_i]. The sign cancels in D[\partial_i\partial_j||]=D[||\partial_i\partial_j]=-D[\partial_i||\partial_j], which we can define to be a metric g_{ij}=-D[\partial_i||\partial_j], if always positive.

The absolute derivative of g_{ij} along \partial_i yields candidates for dual connections \partial_ig_{jk}=-D[\partial_i\partial_j||\partial_k]-D[\partial_j||\partial_i\partial_k]=\Gamma_{ij,k}+\Gamma_{ik,j}^*. This metric and the connections relate to the Taylor series expansion D(p||q) for the first parameter or second parameter. Here for the first parameter:


\begin{align}
&D[p||q]=\frac{1}{2}g_{ij}(q)\Delta\xi^i\Delta\xi^j+\frac{1}{6}h_{ijk}\Delta\xi^i\Delta\xi^j\Delta\xi^k +o(||\Delta\xi||^3)\\
&h_{ijk}=D[\partial_i\partial_j\partial_k||]\\
&\partial_ig_{jk}=\partial_iD[\partial_j\partial_k||]=D[\partial_i\partial_j\partial_k||]+D[\partial_j\partial_k||\partial_i]=h_{ijk}-\Gamma_{jk,i}\\
&h_{ijk}=\partial_ig_{jk}+\Gamma_{jk,i} .
\end{align}

The term D(p||q) is called the divergence or contrast function. A good choice is D(p||q)=\sum p f(\frac{q}{p}) with f convex for u>0. From Jensen's inequality it follows that D(p||q))\geq f\sum p\frac{q}{p}=f(1) and, for f(u)=u\log u, we have

D(p||q)=D^{(-1)}(p||q)=D^{(1)}(q||p)=\sum{p\log\frac{p}{q}},

which is the Kullback-Leibler divergence or relative entropy applicable to the \pm 1-families. In the above,

g_{ij}=-D[\partial_i||\partial_j]=-\partial'_j\partial_i\sum{p(\log;p-\log;q)}=\sum\frac{\partial_ip\partial_jq}{q}=\sum\partial_ip\partial_j\log q=[p=q]
=\sum{p\partial_i\ell\partial_j\ell}

is the Fisher metric. For \alpha\neq\pm 1 a different f yields

D^{(\alpha)}=\frac{4}{1-\alpha^2}(1-\sum p^{\frac{1-\alpha}{2}} p^{\frac{1+\alpha}{2}}).

The Hellinger distance applicable to the 0-family is

D^{(0)}(p||q)=2\sum{(\sqrt(p)-\sqrt(q))^2}=4(1-\sum{\sqrt{pq}}).

In this case, -D^{(0)}[\partial_i||\partial_j] also evaluates to the Fisher metric.

Canonical divergence

We now consider two manifolds S and S^*, represented by two sets of coordinate functions [\theta^i] and [\eta_j]. The corresponding tangent space basis vectors will be denoted by \partial_i=\frac{\partial}{\partial\theta^i} and \partial^i=\frac{\partial}{\partial\eta_i}. The bilinear map \langle,\rangle:TS\times TS^*\rightarrow\mathbb{R} associates a quantity \geq 0 to the dual base vectors. This defines an affine connection \nabla for S and affine connection \nabla^* for S^* that keep \langle X,X^*\rangle constant for parallel transport of X\in TS and X^*\in TS^*, defined through \nabla and \nabla^*.

If S is flat, then there exists a coordinate system \partial_i, that does not change over S. In order to keep \langle\partial_i,\partial^j\rangle constant, \partial^j must not change either, i.e. S^* is also flat. Furthermore, in this case, we can choose coordinate systems such that


\langle\partial_i,\partial^j\rangle=\delta_i^j

If S^* results as a function f on S, then making \eta_i=\theta^i\circ f^{-1}, both coordinate system function sets describe S. The connections are such, though, that \nabla makes S flat and \nabla^* makes S^* flat. This dual space is denoted as (S,g,\nabla,\nabla^*).

This naturally leads to the following definition of a canonical divergence:


D(p||q)=\psi(p)+\phi(q)-\theta^i(p)\eta_i(q)

Note the summation that is a representation of the metric due to \langle\partial_i,\partial^j\rangle=\delta_i^j.

Properties of divergence

The meaning of the canonical divergence depends on the meaning of the metric \langle\partial_i,\partial^j\rangle=\delta_i^j and vice versa ( g_{ij}=-D[\partial_i||\partial_j] ). For the \alpha=\pm 1 metric (Fisher metric) with the dual connections this is the relative entropy. For the self-dual Euclidean space \psi=\phi=\frac{1}{2}\sum{(\theta^i)^2} leads to D(p||q)=\frac{1}{2}\sum(\theta^i(p)-\theta^i(q))^2=\frac{1}{2}d(p,q)^2

Similar to the Euclidean space the following holds:

For an autoparallel sub-manifold parallel transport in it can be expressed with the sub-manifold's base vectors, i.e. \nabla_{\partial_a}\partial_b=\Gamma_{ab}^{c}\partial_c. A one-dimensional autoparallel sub-manifold is a geodesic.

Canonical divergence for the exponential family

For the exponential family p(v;\theta)=\exp[C(v)+\theta^iF_i(v)-\psi(\theta)] one has \exp[\psi(\theta)]=\sum (C(v)+\theta^iF_i). Applying \partial_i on both sides yields \eta_i(\theta)=\partial_i\psi(\theta)=\sum F_i(v)p(v;\theta)=E[F_i]. The other potential \phi(\theta)=\theta^i\eta_i(\theta)-\psi(\theta)=\theta^i\sum p F_i-\psi=E[\log;p-C]=-H(p)-E[C] ( H is entropy, \theta^i F_i=\log p-C(v)+\psi(\theta) and E[\psi(\theta)]=\psi(\theta) was used). g_{ij}=E[\partial_i\ell\partial_j\ell]=E[(F_i-\eta_i)(F_j-\eta_j)]=V[\eta] is the covariance of \eta, the Cramér–Rao bound, i.e. an efficient estimator must be exponential.

The canonical divergence is given by the Kullback-Leibler divergence D(p||q)=\sum p(\log p-\log q) and the triangulation is D(p||q)+D(q||r)-D(p||r)=\sum(p-q)(\log;p-\log;q).

The minimal divergence to a sub-manifold given by a restriction like some constant \eta_i means maximizing H(p)+E[C]. With C=0 this corresponds to the maximum entropy principle.

Canonical divergence for general alpha families

For general \alpha-affine manifolds with \ell^{(\alpha)}=C(v)+\theta^iF_i one has:


\begin{align}
&\eta_i=\sum F_i\ell^{(-\alpha)}\\
&\partial_j\eta_i=g_{ij}=\sum{\partial_i\ell^{(\alpha)}\partial_j\ell^{(-\alpha)}}=\sum F_i\partial_j\ell^{(-\alpha)}\\
&\Psi^{(\alpha\neq -1)}(\theta)=\frac{2}{1+\alpha}\sum p\\
&\Psi^{(\alpha=-1)}(\theta)=\sum p(\log p-1)\\
&\psi(\theta)=\Psi^{(\alpha)}\\
&\phi(\theta)=\Psi^{(-\alpha)}-\sum C(x)\ell^{(-\alpha)}\\
&D^{\alpha}(p||q)=\Psi^{(\alpha)}+\Psi^{(-\alpha)}-\sum\ell_p^{(\alpha)}\ell_q^{(-\alpha)}\\
&D^{\alpha\neq\pm 1}(p||q)=\frac{4}{1-\alpha^2}\sum\{\frac{1-\alpha}{2}p+\frac{1+\alpha}{2}q-p^{\frac{1-\alpha}{2}}q^{\frac{1+\alpha}{2}}\}\\
&D^{\alpha=\pm 1}(p||q)=\sum \{p-q+p\log\frac{p}{q}\}\\
&\theta^i\eta'_i=\sum\{\ell^{(\alpha)}(v;\theta)-C(v)\}\ell^{(-\alpha)}(v;\theta')\\
&D(\theta||\theta')=\psi(\theta)+\phi(\theta)-\theta^i\eta'_i
\end{align}

The connection induced by the divergence is not flat unless \alpha=\pm 1. Then the Pythagorean theorem for two curves intersecting orthogonally at q is:


D^{(\alpha)}(p||r)=D^{(\alpha)}(p||q)+D^{(\alpha)}(q||r)-\frac{1-\alpha^2}{4}D^{(\alpha)}(p||q)D^{(\alpha)}(q||r)

History

The history of information geometry is associated with the discoveries of at least the following people, and many others

Applications

Information geometry can be applied where parametrized distributions play a role.

Here an incomplete list:

See also

References

  1. 1 2 Shun'ichi Amari, Hiroshi Nagaoka - Methods of information geometry, Translations of mathematical monographs; v. 191, American Mathematical Society, 2000 (ISBN 978-0821805312)

Further reading

External links

This article is issued from Wikipedia - version of the Friday, February 26, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.