Goodness of fit

The goodness of fit of a statistical model describes how well it fits a set of observations. Measures of goodness of fit typically summarize the discrepancy between observed values and the values expected under the model in question. Such measures can be used in statistical hypothesis testing, e.g. to test for normality of residuals, to test whether two samples are drawn from identical distributions (see Kolmogorov–Smirnov test), or whether outcome frequencies follow a specified distribution (see Pearson's chi-squared test). In the analysis of variance, one of the components into which the variance is partitioned may be a lack-of-fit sum of squares.

Fit of distributions

In assessing whether a given distribution is suited to a data-set, the following tests and their underlying measures of fit can be used:

• Kolmogorov–Smirnov test;
• Cramér–von Mises criterion;
• Anderson–Darling test;
• Shapiro–Wilk test;
• Chi-squared test;
• Akaike information criterion;
• Hosmer–Lemeshow test;

Regression analysis

In regression analysis, the following topics relate to goodness of fit:

• Coefficient of determination (The R squared measure of goodness of fit);
• Lack-of-fit sum of squares.

Example

One way in which a measure of goodness of fit statistic can be constructed, in the case where the variance of the measurement error is known, is to construct a weighted sum of squared errors:

$\chi^2 = \sum {\frac{(O - E)^2}{\sigma^2}}$

where $\sigma^2$ is the known variance of the observation, O is the observed data and E is the theoretical data.^[1] This definition is only useful when one has estimates for the error on the measurements, but it leads to a situation where a chi-squared distribution can be used to test goodness of fit, provided that the errors can be assumed to have a normal distribution.

The reduced chi-squared statistic is simply the chi-squared divided by the number of degrees of freedom:^[1][2][3][4]

$\chi_\mathrm{red}^2 = \frac{\chi^2}{\nu} = \frac{1}{\nu} \sum {\frac{(O - E)^2}{\sigma^2}}$

where $\nu$ is the number of degrees of freedom, usually given by $N-n$, where $N$ is the number of observations, and $n$ is the number of fitted parameters. The advantage of the reduced chi-squared is that it already normalizes for the number of data points and model complexity. This is also known as the mean square weighted deviation.

As a rule of thumb (again valid only when the variance of the measurement error is known a priori rather than estimated from the data), a $\chi_\mathrm{red}^2 \gg 1$ indicates a poor model fit. A $\chi_\mathrm{red}^2 > 1$ indicates that the fit has not fully captured the data (or that the error variance has been underestimated). In principle, a value of $\chi_\mathrm{red}^2 = 1$ indicates that the extent of the match between observations and estimates is in accord with the error variance. A $\chi_\mathrm{red}^2 < 1$ indicates that the model is 'over-fitting' the data: either the model is improperly fitting noise, or the error variance has been overestimated.^[5]

Categorical data

The following are examples that arise in the context of categorical data.

Pearson's chi-squared test

Pearson's chi-squared test uses a measure of goodness of fit which is the sum of differences between observed and expected outcome frequencies (that is, counts of observations), each squared and divided by the expectation:

$\chi^2 = \sum_{i=1}^n {\frac{(O_i - E_i)}{E_i}^2}$

where:

O_i = an observed frequency (i.e. count) for bin i

E_i = an expected (theoretical) frequency for bin i, asserted by the null hypothesis.

The expected frequency is calculated by:

$E_i \, = \, \bigg( F(Y_u) \, - \, F(Y_l) \bigg) \, N$

where:

F = the cumulative Distribution function for the distribution being tested.

Y_u = the upper limit for class i,

Y_l = the lower limit for class i, and

N = the sample size

The resulting value can be compared to the chi-squared distribution to determine the goodness of fit. In order to determine the degrees of freedom of the chi-squared distribution, one takes the total number of observed frequencies and subtracts the number of estimated parameters. The test statistic follows, approximately, a chi-square distribution with (k − c) degrees of freedom where k is the number of non-empty cells and c is the number of estimated parameters (including location and scale parameters and shape parameters) for the distribution.

Example: equal frequencies of men and women

For example, to test the hypothesis that a random sample of 100 people has been drawn from a population in which men and women are equal in frequency, the observed number of men and women would be compared to the theoretical frequencies of 50 men and 50 women. If there were 44 men in the sample and 56 women, then

$\chi^2 = {(44 - 50)^2 \over 50} + {(56 - 50)^2 \over 50} = 1.44$

If the null hypothesis is true (i.e., men and women are chosen with equal probability in the sample), the test statistic will be drawn from a chi-squared distribution with one degree of freedom. Though one might expect two degrees of freedom (one each for the men and women), we must take into account that the total number of men and women is constrained (100), and thus there is only one degree of freedom (2 − 1). Alternatively, if the male count is known the female count is determined, and vice versa.

Consultation of the chi-squared distribution for 1 degree of freedom shows that the probability of observing this difference (or a more extreme difference than this) if men and women are equally numerous in the population is approximately 0.23. This probability is higher than conventional criteria for statistical significance (.001-.05), so normally we would not reject the null hypothesis that the number of men in the population is the same as the number of women (i.e. we would consider our sample within the range of what we'd expect for a 50/50 male/female ratio.)

Binomial case

A binomial experiment is a sequence of independent trials in which the trials can result in one of two outcomes, success or failure. There are n trials each with probability of success, denoted by p. Provided that np_i ≫ 1 for every i (where i = 1, 2, ..., k), then

$\chi^2 = \sum_{i=1}^{k} {\frac{(N_i - np_i)^2}{np_i}} = \sum_{\mathrm{all\ cells}}^{} {\frac{(\mathrm{O} - \mathrm{E})^2}{\mathrm{E}}}.$

This has approximately a chi-squared distribution with k − 1 degrees of freedom. The fact that there are k − 1 degrees of freedom is a consequence of the restriction $\sum N_i=n$. We know there are k observed cell counts, however, once any k − 1 are known, the remaining one is uniquely determined. Basically, one can say, there are only k − 1 freely determined cell counts, thus k − 1 degrees of freedom.

Other measures of fit

The likelihood ratio test statistic is a measure of the goodness of fit of a model, judged by whether an expanded form of the model provides a substantially improved fit.

See also

• Deviance (statistics) (related to GLM)
• Overfitting

References

1. Laub, Charlie; Kuhl, Tonya L. (n.d.), How Bad is Good? A Critical Look at the Fitting of Reflectivity Models using the Reduced Chi-Square Statistic (PDF), University California, Davis, retrieved 30 May 2015
2. Taylor, John Robert (1997), An introduction to error analysis, University Science Books, p. 268
3. Kirkman, T. W. (n.d.), Chi-Squared Curve Fitting, retrieved 30 May 2015
4. Glover, David M.; Jenkins, William J.; Doney, Scott C. (2008), Least Squares and regression techniques, goodness of fit and tests, non-linear least squares techniques, Woods Hole Oceanographic Institute, archived from the original on July 8, 2008
5. Bevington, Philip R. (1969), Data Reduction and Error Analysis for the Physical Sciences, New York: McGraw-Hill, p. 89, For χ² tests, χ_ν² should be approximately equal to one.