Least-squares function approximation

In mathematics, the idea of least squares can be applied to approximating a given function by a weighted sum of other functions. The best approximation can be defined as that which minimises the difference between the original function and the approximation; for a least-squares approach the quality of the approximation is measured in terms of the squared differences between the two.

Functional analysis

A generalization to approximation of a data set is the approximation of a function by a sum of other functions, usually an orthogonal set:^[1]

$f(x) \approx f_n (x) = a_1 \phi _1 (x) + a_2 \phi _2(x) + \cdots + a_n \phi _n (x), \$

with the set of functions {$\ \phi _j (x)$} an orthonormal set over the interval of interest, say [a, b]: see also Fejér's theorem. The coefficients {$\ a_j$} are selected to make the magnitude of the difference ||f − f_n ||² as small as possible. For example, the magnitude, or norm, of a function g (x ) over the interval [a, b] can be defined by:^[2]

$\|g\| = \left(\int_a^b g^*(x)g(x) \, dx \right)^{1/2}$

where the ‘*’ denotes complex conjugate in the case of complex functions. The extension of Pythagoras' theorem in this manner leads to function spaces and the notion of Lebesgue measure, an idea of “space” more general than the original basis of Euclidean geometry. The { $\phi_j (x)\$ } satisfy orthonormality relations:^[3]

$\int_a^b \phi _i^* (x)\phi _j (x) \, dx =\delta_{ij},$

where δ_ij is the Kronecker delta. Substituting function f_n into these equations then leads to the n-dimensional Pythagorean theorem:^[4]

$\|f_n\|^2 = |a_1|^2 + |a_2|^2 + \cdots + |a_n|^2. \,$

The coefficients {a_j} making ||f − fn||² as small as possible are found to be:^[1]

$a_j = \int_a^b \phi _j^* (x)f (x) \, dx.$

The generalization of the n-dimensional Pythagorean theorem to infinite-dimensional  real inner product spaces is known as Parseval's identity or Parseval's equation.^[5] Particular examples of such a representation of a function are the Fourier series and the generalized Fourier series.

References

1. Cornelius Lanczos (1988). Applied analysis (Reprint of 1956 Prentice–Hall ed.). Dover Publications. pp. 212–213. ISBN 0-486-65656-X.
2. Gerald B Folland (2009). "Equation 3.14". Fourier analysis and its application (Reprint of Wadsworth and Brooks/Cole 1992 ed.). American Mathematical Society Bookstore. p. 69. ISBN 0-8218-4790-2.
3. Folland, Gerald B (2009). Fourier Analysis and Its Applications. American Mathematical Society. p. 69. ISBN 0-8218-4790-2.
4. David J. Saville, Graham R. Wood (1991). "§2.5 Sum of squares". Statistical methods: the geometric approach (3rd ed.). Springer. p. 30. ISBN 0-387-97517-9.
5. Gerald B Folland (2009-01-13). "Equation 3.22". cited work. p. 77. ISBN 0-8218-4790-2.