Consistent estimator

For a broader coverage related to this topic, see Consistency (statistics).
{T1, T2, T3, …} is a sequence of estimators for parameter θ0, the true value of which is 4. This sequence is consistent: the estimators are getting more and more concentrated near the true value θ0; at the same time, these estimators are biased. The limiting distribution of the sequence is a degenerate random variable which equals θ0 with probability 1.

In statistics, a consistent estimator or asymptotically consistent estimator is an estimator—a rule for computing estimates of a parameter θ0—having the property that as the number of data points used increases indefinitely, the resulting sequence of estimates converges in probability to θ0. This means that the distributions of the estimates become more and more concentrated near the true value of the parameter being estimated, so that the probability of the estimator being arbitrarily close to θ0 converges to one.

In practice one constructs an estimator as a function of an available sample of size n, and then imagines being able to keep collecting data and expanding the sample ad infinitum. In this way one would obtain a sequence of estimates indexed by n, and consistency is a property of what occurs as the sample size “grows to infinity”. If the sequence of estimates can be mathematically shown to converge in probability to the true value θ0, it is called a consistent estimator; otherwise the estimator is said to be inconsistent.

Consistency as defined here is sometimes referred to as weak consistency. When we replace convergence in probability with almost sure convergence, then the estimator is said to be strongly consistent. Consistency is related to bias; see bias versus consistency.

Definition

Loosely speaking, an estimator Tn of parameter θ is said to be consistent, if it converges in probability to the true value of the parameter:[1]


    \underset{n\to\infty}{\operatorname{plim}}\;T_n = \theta.

A more rigorous definition takes into account the fact that θ is actually unknown, and thus the convergence in probability must take place for every possible value of this parameter. Suppose {pθ: θ ∈ Θ} is a family of distributions (the parametric model), and Xθ = {X1, X2, … : Xi ~ pθ} is an infinite sample from the distribution pθ. Let { Tn(Xθ) } be a sequence of estimators for some parameter g(θ). Usually Tn will be based on the first n observations of a sample. Then this sequence {Tn} is said to be (weakly) consistent if [2]


    \underset{n\to\infty}{\operatorname{plim}}\;T_n(X^{\theta}) = g(\theta),\ \ \text{for all}\ \theta\in\Theta.

This definition uses g(θ) instead of simply θ, because often one is interested in estimating a certain function or a sub-vector of the underlying parameter. In the next example we estimate the location parameter of the model, but not the scale:

Examples

Sample mean of a normal random variable

Suppose one has a sequence of observations {X1, X2, …} from a normal N(μ, σ2) distribution. To estimate μ based on the first n observations, one can use the sample mean: Tn = (X1 + … + Xn)/n. This defines a sequence of estimators, indexed by the sample size n.

From the properties of the normal distribution, we know the sampling distribution of this statistic: Tn is itself normally distributed, with mean μ and variance σ2/n. Equivalently, \scriptstyle (T_n-\mu)/(\sigma/\sqrt{n}) has a standard normal distribution:


    \Pr\!\left[\,|T_n-\mu|\geq\varepsilon\,\right] = 
    \Pr\!\left[ \frac{\sqrt{n}\,\big|T_n-\mu\big|}{\sigma} \geq \sqrt{n}\varepsilon/\sigma \right] = 
    2\left(1-\Phi\left(\frac{\sqrt{n}\,\varepsilon}{\sigma}\right)\right) \to 0

as n tends to infinity, for any fixed ε > 0. Therefore, the sequence Tn of sample means is consistent for the population mean μ.

Establishing consistency

The notion of asymptotic consistency is very close, almost synonymous to the notion of convergence in probability. As such, any theorem, lemma, or property which establishes convergence in probability may be used to prove the consistency. Many such tools exist:


    \Pr\!\big[h(T_n-\theta)\geq\varepsilon\big] \leq \frac{\operatorname{E}\big[h(T_n-\theta)\big]}{\varepsilon},

the most common choice for function h being either the absolute value (in which case it is known as Markov inequality), or the quadratic function (respectively Chebyshev's inequality).


    T_n\ \xrightarrow{p}\ \theta\ \quad\Rightarrow\quad g(T_n)\ \xrightarrow{p}\ g(\theta)
\begin{align}
  & T_n + S_n \ \xrightarrow{p}\ \alpha+\beta, \\
  & T_n   S_n \ \xrightarrow{p}\ \alpha \beta, \\
  & T_n / S_n \ \xrightarrow{p}\ \alpha/\beta, \text{ provided that }\beta\neq0
  \end{align}
\frac{1}{n}\sum_{i=1}^n g(X_i) \ \xrightarrow{p}\ \operatorname{E}[\,g(X)\,]

Bias versus consistency

Bias is related to consistency as follows: a sequence of estimators is consistent if and only if it converges to a value and the bias converges to zero. Consistent estimators are convergent and asymptotically unbiased (hence converge to the correct value): individual estimators in the sequence may be biased, but the overall sequence still consistent, if the bias converges to zero. Conversely, if the sequence does not converge to a value, then it is not consistent, regardless of whether the estimators in the sequence are biased or not.

Unbiased but not consistent

An estimator can be unbiased but not consistent. For example, for an iid sample {x
1
,..., x
n
} one can use T(X) = x
1
as the estimator of the mean E[x]. Note that here the sampling distribution of T is the same as the underlying distribution (for any n, as it ignores all points but the first), so E[T(X)] = E[x] and it is unbiased, but it does not converge to any value.

However, if a sequence of estimators is unbiased and converges to a value, then it is consistent, as it must converge to the correct value.

Biased but consistent

Alternatively, an estimator can be biased but consistent. For example if the mean is estimated by {1 \over n} \sum x_i + {1 \over n} it is biased, but as n \rightarrow \infty, it approaches the correct value, and so it is consistent.

Important examples include the sample variance and sample standard deviation. Without Bessel's correction (using the sample size n instead of the degrees of freedom n  1), these are both negatively biased but consistent estimators. With the correction, the unbiased sample variance is unbiased, while the corrected sample standard deviation is still biased, but less so, and both are still consistent: the correction factor converges to 1 as sample size grows.

See also

Notes

  1. Amemiya 1985, Definition 3.4.2.
  2. Lehman & Casella 1998, p. 332.
  3. Amemiya 1985, equation (3.2.5).
  4. Amemiya 1985, Theorem 3.2.6.
  5. Amemiya 1985, Theorem 3.2.7.
  6. Newey & McFadden 1994, Chapter 2.

References

External links

This article is issued from Wikipedia - version of the Friday, December 25, 2015. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.