Projection (linear algebra)

"Orthogonal projection" redirects here. For the technical drawing concept, see Orthographic projection. For a concrete discussion of orthogonal projections in finite-dimensional linear spaces, see Vector projection.

In linear algebra and functional analysis, a projection is a linear transformation P from a vector space to itself such that P 2 = P. That is, whenever P is applied twice to any value, it gives the same result as if it were applied once (idempotent). It leaves its image unchanged.[1] Though abstract, this definition of "projection" formalizes and generalizes the idea of graphical projection. One can also consider the effect of a projection on a geometrical object by examining the effect of the projection on points in the object.

Simple example

Orthogonal projection

For example, the function which maps the point (x, y, z) in three-dimensional space R3 to the point (x, y, 0) is an orthogonal projection onto the xy plane. This function is represented by the matrix

 P = \begin{bmatrix} 1 & 0 & 0 \\  0 & 1 & 0 \\  0 & 0 & 0 \end{bmatrix}.

The action of this matrix on an arbitrary vector is

 P \begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix}
x \\ y \\  0 \end{pmatrix}.

To see that P is indeed a projection, i.e., P = P2, we compute

 P^2 \begin{pmatrix} x \\ y \\ z \end{pmatrix} = P \begin{pmatrix} x \\ y \\ 0 \end{pmatrix} = \begin{pmatrix} x \\ y \\  0 \end{pmatrix} = P\begin{pmatrix} x \\ y \\ z \end{pmatrix}.

Oblique projection

A simple example of a non-orthogonal (oblique) projection (for definition see below) is

 P = \begin{bmatrix} 0 & 0  \\  \alpha & 1  \end{bmatrix}.

Via matrix multiplication, one sees that

 P^2 = \begin{bmatrix} 0 & 0  \\  \alpha & 1  \end{bmatrix} \begin{bmatrix} 0 & 0  \\  \alpha & 1  \end{bmatrix}
= \begin{bmatrix} 0 & 0  \\  \alpha & 1  \end{bmatrix} = P.

proving that P is indeed a projection.

The projection P is orthogonal if and only if α = 0.

Properties and classification

Let W be a finite dimensional vector space and P be a projection on W. Suppose the subspaces U and V are the range and kernel of P respectively. Then P has the following basic properties:

  1. By definition, P is idempotent (i.e. P^2 = P).
  2. P is the identity operator I on U
    \forall x \in U: Px = x.
  3. We have a direct sum W = U \oplus V. Every vector x \in W may be decomposed uniquely as x = u + v with u = Px and v = x - Px = (I-P)x, and where u \in U, v \in V.

The range and kernel of a projection are complementary, as are P and Q = I - P. The operator Q is also a projection and the range and kernel of P become the kernel and range of Q and vice versa. We say P is a projection along V onto U (kernel/range) and Q is a projection along U onto V.

In infinite dimensional vector spaces spectrum of a projection is contained in {0, 1}, as

 (\lambda I - P)^{-1}= \frac 1 \lambda I+\frac 1{\lambda(\lambda-1)} P.

Only 0 and 1 can be an eigenvalue of a projection. The corresponding eigenspaces are (respectively) the kernel and range of the projection. Decomposition of a vector space into direct sums is not unique in general. Therefore, given a subspace V, there may be many projections whose range (or kernel) is V.

If a projection is nontrivial it has minimal polynomial x^2-x = x(x-1), which factors into distinct roots, and thus P is diagonalizable.

The product of projections is not, in general, a projection, even if they are orthogonal. If projections commute, then their product is a projection.

Orthogonal projections

When the vector space W has an inner product and is complete (is a Hilbert space) the concept of orthogonality can be used. An orthogonal projection is a projection for which the range U and the null space V are orthogonal subspaces. Thus, for every x and y in W,  \langle Px, (y-Py) \rangle = \langle (x-Px) , Py \rangle = 0. Equivalently:

 \langle x, Py \rangle = \langle Px, Py \rangle = \langle Px, y \rangle .

A projection is orthogonal if and only if it is self-adjoint. Using the self-adjoint and idempotent properties of P, for any x and y in W we have PxU, yPyV, and

 \langle Px, y-Py \rangle = \langle P^2x, y-Py \rangle = \langle Px, P(I-P)y \rangle = \langle Px, (P-P^2)y \rangle = 0 \,

where \langle\cdot,\cdot\rangle is the inner product associated with W. Therefore, Px and yPy are orthogonal.[2] The other direction, namely that if P is orthogonal then it is self-adjoint, follows from

 \langle x, Py \rangle = \langle Px, y \rangle = \langle x, P^*y \rangle

for every x and y in W; thus P = P*.

Properties and special cases

An orthogonal projection is a bounded operator. This is because for every v in the vector space we have, by Cauchy–Schwarz inequality:

\|Pv\|^2 = \langle Pv,Pv\rangle = \langle Pv,v\rangle \le \|Pv\|\cdot\|v\|

Thus \|Pv\|\le \|v\|.

For finite dimensional complex or real vector spaces, the standard inner product can be substituted for \langle\cdot,\cdot\rangle.

Formulas

A simple case occurs when the orthogonal projection is onto a line. If u is a unit vector on the line, then the projection is given by the outer product

 P_u = u u^\mathrm{T}. \,

(If u is complex-valued, the transpose in the above equation is replaced by a Hermetian transpose). This operator leaves u invariant, and it annihilates all vectors orthogonal to u, proving that it is indeed the orthogonal projection onto the line containing u.[3] A simple way to see this is to consider an arbitrary vector x as the sum of a component on the line (i.e. the projected vector we seek) and another perpendicular to it, x=x_\parallel+x_\perp. Applying projection, we get

P_ux=u u^\mathrm{T}x_\parallel+uu^\mathrm{T}x_\perp=u(\mathrm{sign}(u^\mathrm{T}x_\parallel)\|x_\parallel\|)+u\cdot 0=x_\parallel

by the properties of the dot product of parallel and perpendicular vectors.

This formula can be generalized to orthogonal projections on a subspace of arbitrary dimension. Let u1, ..., uk be an orthonormal basis of the subspace U, and let A denote the n-by-k matrix whose columns are u1, ..., uk. Then the projection is given by

P_A = A A^\mathrm{T}[4]

which can be rewritten as

P_A = \sum_i \langle u_i,\cdot\rangle u_i.

The matrix AT is the partial isometry that vanishes on the orthogonal complement of U and A is the isometry that embeds U into the underlying vector space. The range of PA is therefore the final space of A. It is also clear that A·AT is the identity operator on U.

The orthonormality condition can also be dropped. If u1, ..., uk is a (not necessarily orthonormal) basis, and A is the matrix with these vectors as columns, then the projection is

P_A = A (A^\mathrm{T} A)^{-1} A^\mathrm{T}.[5][6]

The matrix A still embeds U into the underlying vector space but is no longer an isometry in general. The matrix (ATA)−1 is a "normalizing factor" that recovers the norm. For example, the rank-1 operator uuT is not a projection if ||u|| ≠ 1. After dividing by uTu = ||u||2, we obtain the projection u(uTu)−1uT onto the subspace spanned by u.

When the range space of the projection is generated by a frame (i.e. the number of generators is greater than its dimension), the formula for the projection takes the form: P_A = A (A^\mathrm{T} A)^+ A^\mathrm{T}. Here A+ stands for the Moore–Penrose pseudoinverse. This is just one of many ways to construct the projection operator.

If a matrix \, [A \ B] is non-singular and AT B = 0 (i.e., B is the null space matrix of A),[7] the following holds:


\begin{align}I&=[A \ B][A \ B]^{-1}\begin{bmatrix}A^\mathrm{T}\\B^\mathrm{T}\end{bmatrix}^{-1} \begin{bmatrix}A^\mathrm{T}\\B^\mathrm{T}\end{bmatrix}\\
&=[A \ B]\left(\begin{bmatrix}A^\mathrm{T}\\B^\mathrm{T}\end{bmatrix}[A \ B]\right)^{-1}\begin{bmatrix}A^\mathrm{T}\\B^\mathrm{T}\end{bmatrix}\\
&=[A \ B]\begin{bmatrix}A^\mathrm{T}A&O\\O&B^\mathrm{T}B\end{bmatrix}^{-1}\begin{bmatrix}A^\mathrm{T}\\B^\mathrm{T}\end{bmatrix}\\
&=A(A^\mathrm{T}A)^{-1}A^\mathrm{T}+B(B^\mathrm{T}B)^{-1}B^\mathrm{T}\end{align}
.

If the orthogonal condition is enhanced to AT W B = AT WT B = 0 with W being non-singular, the following holds:

I = \begin{bmatrix}A & B\end{bmatrix} \begin{bmatrix}(A^\mathrm{T} W A)^{-1} A^\mathrm{T} \\ (B^\mathrm{T} W B)^{-1} B^\mathrm{T} \end{bmatrix} W .

All these formulas also hold for complex inner product spaces, provided that the conjugate transpose is used instead of the transpose. Further details on sums of projectors can be found in Banerjee and Roy (2014).[8]

Oblique projections

The term oblique projections is sometimes used to refer to non-orthogonal projections. These projections are also used to represent spatial figures in two-dimensional drawings (see oblique projection), though not as frequently as orthogonal projections.

Oblique projections are defined by their range and null space. A formula for the matrix representing the projection with a given range and null space can be found as follows. Let the vectors u1, ..., uk form a basis for the range of the projection, and assemble these vectors in the n-by-k matrix A. The range and the null space are complementary spaces, so the null space has dimension nk. It follows that the orthogonal complement of the null space has dimension k. Let v1, ..., vk form a basis for the orthogonal complement of the null space of the projection, and assemble these vectors in the matrix B. Then the projection is defined by

 P = A (B^\mathrm{T} A)^{-1} B^\mathrm{T}.

This expression generalizes the formula for orthogonal projections given above.[9][10]

Canonical forms

Any projection P = P2 on a vector space of dimension d over a field is a diagonalizable matrix, since its minimal polynomial is x2x, which splits into distinct linear factors. Thus there exists a basis in which P has the form

P = I_r\oplus 0_{d-r}

where r is the rank of P. Here Ir is the identity matrix of size r, and 0dr is the zero matrix of size dr. If the vector space is complex and equipped with an inner product, then there is an orthonormal basis in which the matrix of P is[11]

P = \begin{bmatrix}1&\sigma_1 \\ 0&0\end{bmatrix} \oplus \cdots \oplus \begin{bmatrix}1&\sigma_k \\ 0&0\end{bmatrix} \oplus I_m \oplus 0_s .

where σ1σ2 ≥ ... ≥ σk > 0. The integers k, s, m and the real numbers \sigma_i are uniquely determined. Note that 2k + s + m = d. The factor Im ⊕ 0s corresponds to the maximal invariant subspace on which P acts as an orthogonal projection (so that P itself is orthogonal if and only if k = 0) and the σi-blocks correspond to the oblique components.

Projections on normed vector spaces

When the underlying vector space X is a (not necessarily finite-dimensional) normed vector space, analytic questions, irrelevant in the finite-dimensional case, need to be considered. Assume now X is a Banach space.

Many of the algebraic notions discussed above survive the passage to this context. A given direct sum decomposition of X into complementary subspaces still specifies a projection, and vice versa. If X is the direct sum X = U \oplus V, then the operator defined by P(u+v) = u is still a projection with range U and kernel V. It is also clear that P^2 = P. Conversely, if P is projection on X, i.e. P^2 = P, then it is easily verified that (1-P)^2 = (1-P). In other words, 1 - P is also a projection. The relation P^2 = P implies 1 = P + (1-P) and X is the direct sum \mathrm{ran}(P) \oplus \mathrm{ran}(1 - P).

However, in contrast to the finite-dimensional case, projections need not be continuous in general. If a subspace U of X is not closed in the norm topology, then projection onto U is not continuous. In other words, the range of a continuous projection P must be a closed subspace. Furthermore, the kernel of a continuous projection (in fact, a continuous linear operator in general) is closed. Thus a continuous projection P gives a decomposition of X into two complementary closed subspaces: X = \mathrm{ran}(P) \oplus \mathrm{ker}(P) = \mathrm{ker}(1-P) \oplus \mathrm{ker}(P).

The converse holds also, with an additional assumption. Suppose U is a closed subspace of X. If there exists a closed subspace V such that X = UV, then the projection P with range U and kernel V is continuous. This follows from the closed graph theorem. Suppose xnx and Pxny. One needs to show that Px = y. Since U is closed and {Pxn} ⊂ U, y lies in U, i.e. Py = y. Also, xnPxn = (IP)xnxy. Because V is closed and {(IP)xn} ⊂ V, we have xyV, i.e. P(xy) = PxPy = Pxy = 0, which proves the claim.

The above argument makes use of the assumption that both U and V are closed. In general, given a closed subspace U, there need not exist a complementary closed subspace V, although for Hilbert spaces this can always be done by taking the orthogonal complement. For Banach spaces, a one-dimensional subspace always has a closed complementary subspace. This is an immediate consequence of Hahn–Banach theorem. Let U be the linear span of u. By Hahn–Banach, there exists a bounded linear functional φ such that φ(u) = 1. The operator P(x) = φ(x)u satisfies P2 = P, i.e. it is a projection. Boundedness of φ implies continuity of P and therefore ker(P) = ran(IP) is a closed complementary subspace of U.

Applications and further considerations

Projections (orthogonal and otherwise) play a major role in algorithms for certain linear algebra problems:

As stated above, projections are a special case of idempotents. Analytically, orthogonal projections are non-commutative generalizations of characteristic functions. Idempotents are used in classifying, for instance, semisimple algebras, while measure theory begins with considering characteristic functions of measurable sets. Therefore, as one can imagine, projections are very often encountered in the context operator algebras. In particular, a von Neumann algebra is generated by its complete lattice of projections.

Generalizations

More generally, given a map between normed vector spaces T\colon V \to W, one can analogously ask for this map to be an isometry on the orthogonal complement of the kernel: that (\ker T)^\perp \to W be an isometry (compare Partial isometry); in particular it must be onto. The case of an orthogonal projection is when W is a subspace of V. In Riemannian geometry, this is used in the definition of a Riemannian submersion.

See also

Notes

  1. Meyer, pp 386+387
  2. Meyer, p. 433
  3. Meyer, p. 431
  4. Meyer, equation (5.13.4)
  5. Banerjee, Sudipto; Roy, Anindya (2014), Linear Algebra and Matrix Analysis for Statistics, Texts in Statistical Science (1st ed.), Chapman and Hall/CRC, ISBN 978-1420095388
  6. Meyer, equation (5.13.3)
  7. See also Linear least squares (mathematics) § Properties of the least-squares estimators.
  8. Banerjee, Sudipto; Roy, Anindya (2014), Linear Algebra and Matrix Analysis for Statistics, Texts in Statistical Science (1st ed.), Chapman and Hall/CRC, ISBN 978-1420095388
  9. Banerjee, Sudipto; Roy, Anindya (2014), Linear Algebra and Matrix Analysis for Statistics, Texts in Statistical Science (1st ed.), Chapman and Hall/CRC, ISBN 978-1420095388
  10. Meyer, equation (7.10.39)
  11. Doković, D. Ž. (August 1991). "Unitary similarity of projectors". Aequationes Mathematicae 42 (1): 220–224. doi:10.1007/BF01818492.

References

External links

This article is issued from Wikipedia - version of the Wednesday, April 13, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.