Inner products, orthogonality, and projection - Basic Exercises

These exercises use the inner product内積ないせき to measure length and angle, and they check orthogonalization, projection, and least squares both computationally and conceptually.

1Exercise plan

An inner product内積ないせき adds lengths and angles to a vector space. A projection射影しゃえい drops an unreachable point to the closest point inside a subspace部分空間ぶぶんくうかん. Before computing, check orthogonality and whether a vector used in a denominator is nonzero.

In these exercises, compute the inner product内積ないせき before drawing geometric conclusions. Orthogonality means the inner product is zero, the norm comes from $\left|\right|x|{|}^2=<x,x>$||x||^2=<x,x>, and projection coefficients come from dividing by the squared norm of the direction vector.

For projection and least-squares problems, the residual is the diagnostic. If $p$p is the projection of $x$x onto a subspace, then $x-p$x-p must be orthogonal to every vector in that subspace. For a least-squares solution of $Ax\approx b$Ax\approx b, this becomes $A^T(b-Ax)=0$A^T(b-Ax)=0, equivalently the normal equations $A^{T}Ax=A^{T}b$A^TAx=A^Tb.

2Problem 1

For $u=(1,2{)}^T$u=(1,2)^T and $v=(3,-1{)}^T$v=(3,-1)^T, find $⟨u,v⟩$\langle u,v\rangle, $\parallel u\parallel$\|u\|, and $\parallel v\parallel$\|v\|, and decide whether $u$u and $v$v are orthogonal直交ちょっこう.

2.1Answer example

and

Since $⟨u,v⟩\ne 0$\langle u,v\rangle\ne0, they are not orthogonal.

2.2Explanation

Orthogonality is judged by the inner product, not by the visual appearance of components.

3Problem 2

For $a=(1,0{)}^T$a=(1,0)^T and $b=(2,0{)}^T$b=(2,0)^T, confirm $\parallel a+b\parallel =\parallel a\parallel +\parallel b\parallel$\|a+b\|=\|a\|+\|b\| and explain equality.

3.1Answer example

while $\parallel a\parallel =1$\|a\|=1 and $\parallel b\parallel =2$\|b\|=2. Equality holds because the vectors point in the same direction.

3.2Explanation

This is the equality case of the triangle inequality三角不等式さんかくふとうしき.

4Problem 3

Apply Gram-Schmidt orthogonalization直交化ちょっこうか to

4.1Answer example

Let $u_1=x_1=(1,1,0{)}^T$u_1=x_1=(1,1,0)^T. Then

4.2Explanation

Because $⟨u_1,u_1⟩=2\ne 0$\langle u_1,u_1\rangle=2\ne0, division is allowed. Orthogonalization subtracts the component in the $u_1$u_1 direction.

5Problem 4

For $y=(2,1{)}^T$y=(2,1)^T and $u=(1,1{)}^T$u=(1,1)^T, find the orthogonal projection of $y$y onto $\mathrm{span}\left(u\right)$\operatorname{span}(u).

5.1Answer example

5.2Explanation

The formula cannot be used for $u=0$u=0 because the denominator would be 0.

6Problem 5

For $A=(1,1{)}^T$A=(1,1)^T and $b=(2,0{)}^T$b=(2,0)^T, find the least-squares solution最小二乗解さいしょうにじょうかい of $Ax=b$Ax=b and explain the projection relation.

6.1Answer example

The normal equation正規方程式せいきほうていしき is $A^{T}Ax=A^{T}b$A^TAx=A^Tb. Since $A^{T}A=2$A^TA=2 and $A^{T}b=2$A^Tb=2, $x=1$x=1. The approximating vector is

6.2Explanation

The vector $b$b is not in $\mathrm{Col}\left(A\right)$\operatorname{Col}(A), so the least-squares method chooses the closest point in the column space.

7Problem 6

In $R^2$\mathbb R^2, define

Check that they are norms.

7.1Answer example

Both are nonnegative and vanish only at $0$0. For a scalar $c$c,

Also,

by the scalar triangle inequality, and for $i=1,2$i=1,2,

so $\parallel x+y{\parallel }_{\mathrm{\infty }}\text{[PARSE ERROR: Undefined("Command("le")")]}\parallel x{\parallel }_{\mathrm{\infty }}+\parallel y{\parallel }_{\mathrm{\infty }}$\|x+y\|_\infty\le\|x\|_\infty+\|y\|_\infty.

7.2Explanation

A norm is any length function satisfying nonnegativity, homogeneity同次性どうじせい, and the triangle inequality.

8Problem 7

For $u=(1,2{)}^T$u=(1,2)^T, $v=(2,4{)}^T$v=(2,4)^T, and $w=(1,-1{)}^T$w=(1,-1)^T, decide when equality holds in Cauchy-Schwarz for $(u,v)$(u,v) and $(u,w)$(u,w).

8.1Answer example

Since $v=2u$v=2u, equality holds for $(u,v)$(u,v). Since $w$w is not a scalar multiple of $u$u, equality does not hold for $(u,w)$(u,w).

8.2Explanation

Equality in Cauchy-Schwarz is not just a numerical accident. It occurs exactly when the two vectors lie on the same line, that is, when they are linearly dependent. Here $v=2u$v=2u, so equality holds for $(u,v)$(u,v); $w$w is not a scalar multiple of $u$u, so equality fails for $(u,w)$(u,w).

9Problem 8

Let

Find the projection of $y=(1,2,3{)}^T$y=(1,2,3)^T onto $U$U and a basis of $U^{\perp }$U^\perp.

9.1Answer example

Let $u_1=(1,1,0{)}^T$u_1=(1,1,0)^T, $u_2=(1,0,1{)}^T$u_2=(1,0,1)^T, and $p=a{u}_1+b{u}_2$p=au_1+bu_2. The condition $y-p\perp u_1,u_2$y-p\perp u_1,u_2 gives

Thus $a=\frac{2}{3}$a=\frac23 and $b=\frac{5}{3}$b=\frac53, so

A vector orthogonal to both generators is $n=(1,-1,-1{)}^T$n=(1,-1,-1)^T, so $\left\{n\right\}$\{n\} is a basis of $U^{\perp }$U^\perp.

9.2Explanation

Projection onto a subspace is found by making the residual orthogonal to every direction in the subspace.

10Problem 9

For

solve the normal equations and check residual orthogonality.

10.1Answer example

Solving gives $\alpha =\frac{7}{6}$\alpha=\frac76 and $\beta =\frac{1}{2}$\beta=\frac12. The least-squares solution is $(7/6,1/2{)}^T$(7/6,1/2)^T. The residual is

and $A^{T}r=0$A^Tr=0.

10.2Explanation

The residual is orthogonal to the column space exactly when the approximating point is closest.

11Problem 10

For $M=\left(\begin{array}{cc}2& 0\\ 0& 1\end{array}\right)$M=\begin{pmatrix}2&0\\0&1\end{pmatrix}, define $⟨x,y{⟩}_M=x^{T}My$\langle x,y\rangle_M=x^TMy. Confirm this is an inner product.

11.1Answer example

The matrix $M$M is symmetric, and

which is positive for $x\ne 0$x\ne0. Additivity and homogeneity follow from matrix multiplication. Thus it is an inner product.

11.2Explanation

A positive definite symmetric matrix defines a weighted inner product.

12Problem 11

Explain why $x^{T}Ny$x^TNy with $N=\left(\begin{array}{cc}1& 0\\ 0& -1\end{array}\right)$N=\begin{pmatrix}1&0\\0&-1\end{pmatrix} is not an inner product.

12.1Answer example

For $e_2=(0,1{)}^T$e_2=(0,1)^T,

violating positive definiteness.

12.2Explanation

A bilinear-looking expression is not an inner product if squared length can be negative.

13Order note: complex inner products

The next problem is a preview of the later exercises on complex inner products複素内積ふくそないせき and unitary matricesユニタリ行列ぎょうれつ. The only fact needed here is this: because of conjugate symmetry共役対称性きょうやくたいしょうせい, a complex inner product must make $⟨z,z⟩$\langle z,z\rangle a nonnegative real number.

14Problem 12

On $\mathbb{C}$\mathbb C, explain why $(z,w)=zw$(z,w)=zw is not a complex inner product複素内積ふくそないせき.

14.1Answer example

Taking $z=i$z=i gives $(i,i)=i^2=-1$(i,i)=i^2=-1. A complex inner product must have $⟨z,z⟩\text{[PARSE ERROR: Undefined("Command("ge")")]}0$\langle z,z\rangle\ge0 real.

14.2Explanation

Complex inner products need conjugation and conjugate symmetry to make squared lengths nonnegative real numbers.

15Problem 13

For $u=(1,0{)}^T$u=(1,0)^T, $v=(0,1{)}^T$v=(0,1)^T, and $w=(2,0{)}^T$w=(2,0)^T, confirm strict and equality cases of the triangle inequality.

15.1Answer example

while

The first pair points in different directions; the second pair lies on the same positive ray.

This is a boundary case for the triangle inequality三角不等式さんかくふとうしき. Except for zero-vector cases, equality appears when the two vectors are positive scalar multiples of each other. If the directions differ, the broken-line length is longer than the direct distance.

16Related lectures

17Related exercises

18Proof exercise: Cauchy-Schwarz and projection minimality

18.1Problem

In this proof exercise, work in a real inner product space実内積空間じつないせきくうかん. The complex case requires tracking conjugates and is treated in the later complex inner product exercises.

For an inner product space and a nonzero vector $u$u, prove Cauchy-Schwarz and explain why the projection of $y$y onto $\mathrm{span}\left(u\right)$\operatorname{span}(u) minimizes distance to $y$y. Also prove the general subspace statement: if $p\in U$p\in U and $v-p\perp U$v-p\perp U, then $p$p is the point of $U$U closest to $v$v.

18.2Answer

For $u\ne 0$u\ne0, consider

and choose $t=⟨x,u⟩/⟨u,u⟩$t=\langle x,u\rangle/\langle u,u\rangle. This gives

For projection, write

Then $y-p\perp u$y-p\perp u. For any $cu\in \mathrm{span}\left(u\right)$cu\in\operatorname{span}(u),

and the two summands are orthogonal, so

More generally, let $p\in U$p\in U and assume $v-p\perp U$v-p\perp U. For any $q\in U$q\in U,

Since $p-q\in U$p-q\in U, the two summands are orthogonal. Hence

Therefore $p$p is the closest point in $U$U to $v$v.

18.3Explanation

The same orthogonality idea proves both the inequality and the closest-point property of projections, first for a one-dimensional span and then for an arbitrary subspace.

19Supplementary proof problem: least squares and normal equations

19.1Problem 14

For a real matrix $A$A and vector $b$b, prove that $x$x is a least-squares solution of $Ax\simeq b$Ax\simeq b if and only if the residual $r=b-Ax$r=b-Ax is orthogonal to $\mathrm{Col}\left(A\right)$\operatorname{Col}(A), equivalently $A^{T}r=0$A^Tr=0.

19.2Answer example

For any perturbation $h$h,

The squared error is

If $r\perp \mathrm{Col}\left(A\right)$r\perp\operatorname{Col}(A), then $⟨r,Ah⟩=0$\langle r,Ah\rangle=0 for every $h$h, so the error is at least $\parallel r{\parallel }^2$\|r\|^2. Conversely, if $x$x minimizes the error, the one-variable function $\parallel r-tAh{\parallel }^2$\|r-tAh\|^2 has derivative 0 at $t=0$t=0 for every $h$h, so $⟨r,Ah⟩=0$\langle r,Ah\rangle=0. This is equivalent to $A^{T}r=0$A^Tr=0.

19.3Explanation

The normal equation is the algebraic form of the geometric condition that the residual points perpendicular to the column space.