Information projection

In information theory, the information projection or I-projection of a probability distribution q onto a set of distributions P is

p^* = \underset{p \in P}{\arg\min} \operatorname{D}_{\mathrm{KL}}(p||q)

where D_{\mathrm{KL}} is the Kullback–Leibler divergence from p to q. Viewing the Kullback–Leibler divergence as a measure of distance, the I-projection p^* is the "closest" distribution to q of all the distributions in P.

The I-projection is useful in setting up information geometry, notably because of the following inequality:

\operatorname{D}_{\mathrm{KL}}(p||r) \geq \operatorname{D}_{\mathrm{KL}}(p||p^*) + \operatorname{D}_{\mathrm{KL}}(p^*||r)

This inequality can be interpreted as an information-geometric version of Pythagoras' triangle inequality theorem, where KL divergence is viewed as squared distance in a Euclidean space.

It is worthwile to note that since  \operatorname{D}_{\mathrm{KL}}(p||q) \geq 0 and continuous in p, if P is closed and non-empty, then there exists at least one minimizer to the optimization problem framed above. Furthermore if P is convex, then the optimum distribution is unique.

The reverse I-projection is

p^* = \underset{p \in P}{\arg\min} \operatorname{D}_{\mathrm{KL}}(q||p)

See also

References


This article is issued from Wikipedia - version of the Monday, April 11, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.