Observed information

In statistics, the observed information, or observed Fisher information, is the negative of the second derivative (the Hessian matrix) of the "log-likelihood" (the logarithm of the likelihood function). It is a sample-based version of the Fisher information.

Definition

Suppose we observe random variables X_1,\ldots,X_n, independent and identically distributed with density f(X; θ), where θ is a (possibly unknown) vector. Then the log-likelihood of the parameters \theta given the data X_1,\ldots,X_n is

\ell(\theta | X_1,\ldots,X_n) = \sum_{i=1}^n \log f(X_i| \theta) .

We define the observed information matrix at \theta^{*} as

\mathcal{J}(\theta^*) 
  = - \left. 
    \nabla \nabla^{\top} 
    \ell(\theta)
  \right|_{\theta=\theta^*}
= -
\left.
\left( \begin{array}{cccc}
  \tfrac{\partial^2}{\partial \theta_1^2}
  &  \tfrac{\partial^2}{\partial \theta_1 \partial \theta_2}
  &  \cdots
  &  \tfrac{\partial^2}{\partial \theta_1 \partial \theta_n} \\
  \tfrac{\partial^2}{\partial \theta_2 \partial \theta_1}
  &  \tfrac{\partial^2}{\partial \theta_2^2}
  &  \cdots
  &  \tfrac{\partial^2}{\partial \theta_2 \partial \theta_n} \\
  \vdots &
  \vdots &
  \ddots &
  \vdots \\
  \tfrac{\partial^2}{\partial \theta_n \partial \theta_1}
  &  \tfrac{\partial^2}{\partial \theta_n \partial \theta_2}
  &  \cdots
  &  \tfrac{\partial^2}{\partial \theta_n^2} \\
\end{array} \right) 
\ell(\theta)
\right|_{\theta = \theta^*}

In many instances, the observed information is evaluated at the maximum-likelihood estimate.[1]

Fisher information

The Fisher information \mathcal{I}(\theta) is the expected value of the observed information given a single observation X distributed according to the hypothetical model with parameter \theta:

\mathcal{I}(\theta) = \mathrm{E}(\mathcal{J}(\theta)).

Applications

In a notable article, Bradley Efron and David V. Hinkley [2] argued that the observed information should be used in preference to the expected information when employing normal approximations for the distribution of maximum-likelihood estimates.

See also

References

  1. Dodge, Y. (2003) The Oxford Dictionary of Statistical Terms, OUP. ISBN 0-19-920613-9
  2. Efron, B.; Hinkley, D.V. (1978). "Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher Information". Biometrika 65 (3): 457487. doi:10.1093/biomet/65.3.457. JSTOR 2335893. MR 0521817.
This article is issued from Wikipedia - version of the Tuesday, June 18, 2013. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.