Information projection

In information theory, the information projection or I-projection of a probability distribution q onto a set of distributions P is

$p^* = \underset{p \in P}{\arg\min} \operatorname{D}_{\mathrm{KL}}(p||q)$

where $D_{\mathrm{KL}}$ is the Kullback–Leibler divergence from p to q. Viewing the Kullback–Leibler divergence as a measure of distance, the I-projection $p^*$ is the "closest" distribution to q of all the distributions in P.

The I-projection is useful in setting up information geometry, notably because of the following inequality:

$\operatorname{D}_{\mathrm{KL}}(p||r) \geq \operatorname{D}_{\mathrm{KL}}(p||p^*) + \operatorname{D}_{\mathrm{KL}}(p^*||r)$

This inequality can be interpreted as an information-geometric version of Pythagoras' triangle inequality theorem, where KL divergence is viewed as squared distance in a Euclidean space.

It is worthwile to note that since $\operatorname{D}_{\mathrm{KL}}(p||q) \geq 0$ and continuous in p, if P is closed and non-empty, then there exists at least one minimizer to the optimization problem framed above. Furthermore if P is convex, then the optimum distribution is unique.

The reverse I-projection is

$p^* = \underset{p \in P}{\arg\min} \operatorname{D}_{\mathrm{KL}}(q||p)$

See also

• Sanov's theorem

References

• Cover, Thomas M.; Thomas, Joy A. (2006). Elements of Information Theory (2 ed.). Hoboken, New Jersey: Wiley Interscience. p. 362.