Orthogonal complements直交補空間ちょっこうほくうかん and projections射影しゃえい

1Introduction

The key idea of this lecture is to decompose a vector uniquely into a component lying in a subspace部分空間ぶぶんくうかん and a component orthogonal直交ちょっこう to that subspace.

In projection and least squares最小二乗法さいしょうにじょうほう, one seeks the point in the column space列空間れつくうかん closest to a given vector. The central condition is that the error is orthogonal to the column space. The orthogonal complement直交補空間ちょっこうほくうかん organizes this condition.

2Terms and definitions

For a subspace $U$U of an inner product space内積空間ないせきくうかん $V$V, the orthogonal complement直交補空間ちょっこうほくうかん is

The orthogonal projection直交射影ちょっこうしゃえい of $v$v onto $U$U is the component $p$p in the decomposition

3Plan

We first verify that $U^{\perp }$U^\perp is a subspace. Then, in finite-dimensional inner product spaces, we explain the decomposition

and derive the projection formula.

4Intuitive explanation

If $U$U is a line in the plane, any vector $v$v can be split into its shadow $p$p along the line and a perpendicular remainder $r$r. The set of all possible perpendicular remainders is $U^{\perp }$U^\perp.

The component $p$p is important because it is the closest vector in $U$U to $v$v. When the error $v-p$v-p is orthogonal to $U$U, there is no remaining direction inside $U$U along which one can move to get closer.

5Precise explanation

5.11. $U^{\perp }$U^\perp is a subspace

Let $v,w\in U^{\perp }$v,w\in U^\perp and let $a,b$a,b be scalars. For every $u\in U$u\in U,

Therefore $av+bw\in U^{\perp }$av+bw\in U^\perp, so $U^{\perp }$U^\perp is a subspace.

5.22. Orthogonal decomposition

If $V$V is finite-dimensional and $U$U is a subspace, every $v\in V$v\in V can be written uniquely as

For existence, take an orthonormal basis正規直交基底せいきちょっこうきてい $e_1,\dots ,e_k$e_1,\ldots,e_k of $U$U and set

Then $p\in U$p\in U, and for every $j$j,

Hence $v-p\in U^{\perp }$v-p\in U^\perp.

For uniqueness, suppose

Then $p_1-p_2=r_2-r_1$p_1-p_2=r_2-r_1 lies in both $U$U and $U^{\perp }$U^\perp. If $x\in U\cap U^{\perp }$x\in U\cap U^\perp, then $⟨x,x⟩=0$\langle x,x\rangle=0, so $x=0$x=0. Thus $p_1=p_2$p_1=p_2 and $r_1=r_2$r_1=r_2.

Consequently, in finite dimensions,

5.33. Projection formula using an orthonormal basis

If $e_1,\dots ,e_k$e_1,\ldots,e_k is an orthonormal basis of $U$U, then

The reason is that the residual $v-{\mathrm{proj}}_U\left(v\right)$v-\operatorname{proj}_U(v) is orthogonal to every basis vector of $U$U.

In the complex case, the same formula is used with the convention of this series: the first variable is linear, so the coefficient is $⟨v,e_i⟩$\langle v,e_i\rangle. Under the opposite convention, conjugates appear in different positions.

As a matrix, an orthogonal projection $P$P satisfies

The first equation says that projecting twice changes nothing after the first projection; the second says the projection is compatible with the inner product.

6Concrete example

In $R^2$\mathbb R^2, let

With

the projection of $v=(2,0)$v=(2,0) onto $U$U is

The residual is

which is orthogonal to $(1,1)$(1,1).

7Another viewpoint

Geometrically, $U^{\perp }$U^\perp is the collection of directions perpendicular to $U$U. Algebraically, it is the solution space of the equations $⟨v,u⟩=0$\langle v,u\rangle=0 for all $u\in U$u\in U.

8Criteria

• When projection, shortest distance, or residuals appear, identify the relevant orthogonal complement.
• To split a vector into a component inside a subspace and a perpendicular error, use orthogonal decomposition.
• In least squares, the residual belongs to the orthogonal complement of the column space.

9Scope and limitations

In finite-dimensional inner product spaces, orthogonal decomposition is stable. In infinite dimensions, closedness of the subspace matters; a non-closed subspace may fail to have nearest points or projections. In complex spaces, transpose is replaced by conjugate transpose $A^{*}$A^* in projection and least-squares formulas.

10Final forms

11In one sentence

The orthogonal complement直交補空間ちょっこうほくうかん is the space that holds the perpendicular error to a subspace.

The orthogonal projection直交射影ちょっこうしゃえい of $x$x onto a subspace $W$W is the vector $p\in W$p\in W for which the residual $x-p$x-p lies in $W^{\perp }$W^\perp. This orthogonality condition is both the geometric meaning and the computational test.

For least-squares problems, the same idea says that $Ax$Ax may not equal $b$b, so we choose $x$x so that the residual $b-Ax$b-Ax is orthogonal to the column space of $A$A. This gives the normal equations $A^{T}Ax=A^{T}b$A^TAx=A^Tb. The solution is unique when the relevant columns are independent; otherwise the projection is still unique, but the coefficient vector may not be.

12Theorem and proof: orthogonal projection minimizes error

Let $U$U be a subspace of an inner product space, and suppose $p\in U$p\in U satisfies $v-p\perp U$v-p\perp U. Then $p$p is the vector in $U$U closest to $v$v.

For any $u\in U$u\in U,

Since $p-u\in U$p-u\in U and $v-p\perp U$v-p\perp U, the two summands are orthogonal. By the Pythagorean theorem,

Thus the distance is minimized when $u=p$u=p. This is the geometric basis of least squares.

13Exercise link

14Related links