Orthogonal Projection

Section 3.3 Orthogonal Projection

In Exercise 3.1.18, we saw that the Fourier expansion theorem gives us an efficient way of testing whether or not a vector belongs to the span of an orthogonal set. When the answer is “no”, the quantity we compute while testing turns out to be very useful: it gives the orthogonal projection of that vector onto the span of our orthogonal set. This turns out to be exactly the ingredient needed to solve certain minimum distance problems.

We hinted above that the calculations we’ve been doing have a lot to do with projection. Since any single nonzero vector forms an orthogonal basis for its span, the projection

\begin{equation*} \proj{\uu}{\vv}=\left(\frac{\uu\dotp\vv}{\len{\uu}^2}\right)\uu \end{equation*}

can be viewed as the orthogonal projection of the vector \(\vv\text{,}\) not onto the vector \(\uu\text{,}\) but onto the subspace \(\spn\{\uu\}\text{.}\) This is, after all, how we viewed projections in elementary linear algebra: we drop the perpendicular from the tip of \(\vv\) onto the line in the direction of \(\uu\text{.}\)

Now that we know how to define an orthogonal basis for a subspace, we can define orthogonal projection onto subspaces of dimension greater than one.

Definition 3.3.1.

Let \(U\) be a subspace of \(\R^n\) with orthogonal basis \(\{\uu_1,\ldots, \uu_k\}\text{.}\) For any vector \(\vv\in \R^n\text{,}\) we define the orthogonal projection of \(\vv\) onto \(U\) by

\begin{equation*} \proj{U}{\vv} = \sum_{i=1}^k\left(\frac{\uu_i\dotp\vv}{\len{\uu_i}^2}\right)\uu_i\text{.} \end{equation*}

Note that \(\proj{U}{\vv}\) is indeed an element of \(U\text{,}\) since it’s a linear combination of its basis vectors. In the case of the trivial subspace \(U=\{\zer\}\text{,}\) we define orthogonal projection of any vector to be \(\zer\text{,}\) since really, what other choice do we have? (This case isn’t really of any interest, we just like being thorough.)

Let’s see how this might be put to use in a classic problem: finding the distance from a point to a plane.

Example 3.3.2.

Find the distance from the point \((3,1,-2)\) to the plane \(P\) defined by \(x-2y+4z=0\text{.}\)

Solution 1. Using projection onto a normal vector

In an elementary linear algebra (or calculus) course, we would solve this problem as follows. First, we would need two vectors parallel to the plane. If \(\bbm x\\y\\z\ebm\) lies in the plane, then \(x-2y+4z=0\text{,}\) so \(x=2y-4z\text{,}\) and

\begin{equation*} \bbm x\\y\\z\ebm = \bbm 2y-4z\\y\\z\ebm = y\bbm 2\\1\\0\ebm + z\bbm -4\\0\\1\ebm\text{,} \end{equation*}

so \(\uu=\bbm 2\\1\\0\ebm\) and \(\vv\bbm -4\\0\\1\ebm\) are parallel to the plane. We then compute the normal vector

\begin{equation*} \mathbf{n}=\uu\times\vv=\bbm 1\\-2\\4\ebm\text{,} \end{equation*}

and compute the projection of the position vector \(\mathbf{p}=\bbm 3,1,-2\ebm\) for the point \(P=(3,1,-2)\) onto \(\mathbf{n}\text{.}\) This gives the vector

\begin{equation*} \xx = \left(\frac{\mathbf{p}\dotp\mathbf{n}}{\len{\mathbf{n}}^2}\right)\mathbf{n} = \frac{-7}{21}\bbm 1\\-2\\4\ebm =\bbm-1/3\\2/3\\-4/3\ebm\text{.} \end{equation*}

Now, this vector is parallel to \(\mathbf{n}\text{,}\) so it’s perpendicular to the plane. Subtracting it from \(\mathbf{p}\) gives a vector parallel to the plane, and this is the position vector for the point we seek.

\begin{equation*} \mathbf{q}=\mathbf{p}-\xx=\bbm 3\\1\\-2\ebm-\bbm -1/3\\-2/3\\-4/3\ebm = \bbm 10/3\\1/3\\-2/3\ebm \end{equation*}

so the closest point is \(Q=\bigl(\frac{10}{3},\frac13,-\frac{2}{3}\bigr)\text{.}\) We weren’t asked for it, but note that if we wanted the distance from the point \(P\) to the plane, this is given by \(\len{\xx}=\frac13\sqrt{21}\text{.}\)

Solution 2. Using orthogonal projection

Let’s solve the same problem using orthogonal projection. First, we have to deal with the fact that the vectors \(\uu\) and \(\vv\) are probably not orthogonal. To get around this, we replace \(\vv\) with

\begin{equation*} \ww = \vv-\left(\frac{\vv\dotp\uu}{\len{\uu}^2}\right)\uu = \bbm -4\\0\\1\ebm+\frac 85\bbm 2\\1\\0\ebm = \bbm -4/5\\8/5\\1\ebm\text{.} \end{equation*}

We now set

\begin{align*} \mathbf{q} \amp =\left(\frac{\mathbf{p}\dotp\uu}{\len{\uu}^2}\right)\uu-\left(\frac{\mathbf{p}\dotp\ww}{\len{\ww}^2}\right)\ww\\ \amp = \frac{7}{5}\bbm 2\\1\\0\ebm +\frac{-14}{105}\bbm -4\\8\\5\ebm \\ \amp = \bbm 10/3\\1/3\\-2/3\ebm\text{.} \end{align*}

Lo and behold, we get the same answer as before.

The only problem with Definition 3.3.1 is that it appears to depend on the choice of orthogonal basis. To see that it doesn’t, we need one more definition.

Definition 3.3.3.

For any subspace \(U\) of \(\R^n\text{,}\) we define the orthogonal complement of \(U\text{,}\) denoted \(U^\bot\text{,}\) by

\begin{equation*} U^\bot = \{\xx\in\R^n \,|\, \xx\dotp\yy = 0 \text{ for all } \yy\in U\}\text{.} \end{equation*}

The term “complement” comes from terminology we mentioned early on, but didn’t spend much time on. Theorem 1.8.10 told us that for any subspace \(U\) of a vector space \(V\text{,}\) it is possible to construct another subspace \(W\) of \(V\) such that \(V = U\oplus W\text{.}\) The subspace \(W\) is known as a complement of \(U\text{.}\) A complement is not unique, but the orthogonal complement is. As you might guess from the name, \(U^\bot\) is also a subspace of \(\R^n\text{.}\)

Exercise 3.3.4.

Show that \(U^\bot\) is a subspace of \(\R^n\text{.}\)

Hint.

The trusty Subspace Test is your friend here. Just be careful to work correctly with the definition of \(U^\bot\text{.}\)

Theorem 3.3.5. Projection Theorem.

Let \(U\) be a subspace of \(\R^n\text{,}\) let \(\xx\) be any vector in \(\R^n\text{,}\) and let \(\mathbf{p}=\proj{U}{\xx}\text{.}\) Then:

\(\mathbf{p}\in U\text{,}\) and \(\xx-\mathbf{p}\in U^\bot\text{.}\)
\(\mathbf{p}\) is the closest vector in \(U\) to the vector \(\xx\text{,}\) in the sense that the distance \(d(\mathbf{p},\xx)\) is minimal among all vectors in \(U\text{.}\) That is, for all \(\uu\neq \mathbf{p}\in U\text{,}\) we have

\begin{equation*} \len{\xx-\mathbf{p}}\lt\len{\xx-\uu}\text{.} \end{equation*}

Strategy.

For the first part, review the Orthogonal Lemma, and convince yourself that this says the same thing. The second part is the hard part, and it requires a trick: we can write \(\xx-\uu\) as \((\xx-\mathbf{p})+(\mathbf{p}-\uu)\text{,}\) and then notice that \(\mathbf{p}-\uu\in U\text{.}\) What can we say using the first part, and the Pythagorean theorem?

Proof.

By Definition 3.3.1, \(\mathbf{p}\) is a linear combination of elements in \(U\text{,}\) so \(\mathbf{p}\in U\text{.}\) The fact that \(\xx-\mathbf{p}\in U^\bot\) follows directly from the Orthogonal Lemma.

Choose any \(\uu\in U\) with \(\uu \neq \mathbf{p}\text{,}\) and write

\begin{equation*} \xx-\uu = (\xx-\mathbf{p})+(\mathbf{p}-\uu)\text{.} \end{equation*}

Since \(\mathbf{p}-\uu\in U\) and \(\xx-\mathbf{p}\in U^\bot\text{,}\) we know that these two vectors are orthogonal, and therefore,

\begin{equation*} \len{\xx-\uu}^2=\len{\xx-\mathbf{p}}^2+\len{\mathbf{p}-\uu}^2\gt \len{\xx-\mathbf{p}}^2\text{,} \end{equation*}

by the Pythagorean Theorem.

Exercise 3.3.6.

Show that \(U\cap U^\bot = \{\zer\}\text{.}\) Use this fact to show that Definition 3.3.1 does not depend on the choice orthogonal basis.

Hint.

Suppose we find vectors \(\mathbf{p}\) and \(\mathbf{p}'\) using basis \(B\) and \(B'\text{.}\) Note that \(\mathbf{p}-\mathbf{p}'\in U\text{,}\) but also that

\begin{equation*} \mathbf{p}-\mathbf{p}' = (\mathbf{p}-\xx)-(\mathbf{p}'-\xx) \end{equation*}

Now use Theorem 3.3.5.

Finally, we note one more useful fact. The process of sending a vector to its orthogonal projection defines an operator on \(\R^n\text{,}\) and yes, it’s linear.

Theorem 3.3.7.

Let \(U\) be a subspace of \(\R^n\text{,}\) and define a function \(P_U:\R^n\to \R^n\) by

\begin{equation*} P_U(\xx) = \proj{U}{\xx} \text{ for any } \xx\in\R^n\text{.} \end{equation*}

Then \(P_U\) is a linear operator such that \(U=\im P_U\) and \(U^\bot = \ker P_U\text{.}\)

Strategy.

The fact that \(P_U\) is linear follows from properties of the dot product, and some careful checking. We know that \(\im P_U\subseteq U\) by definition of the projection, and you can show that \(P_U\) acts as the identity on \(U\) using the Fourier expansion theorem.

If \(\xx\in U^\bot\text{,}\) then \(P_U(\xx)=\zer\) by definition of \(P_U\text{.}\) (Recall that it is defined using dot products with vectors in \(U\text{.}\)) If \(\xx\in \ker P_U\text{,}\) use the Projection Theorem, to show that \(\xx\in U^\bot\text{.}\)

Remark 3.3.8.

It follows from this result and the Dimension Theorem that

\begin{equation*} \dim U + \dim U^\bot = n\text{,} \end{equation*}

and since \(U\cap U^\bot = \{\zer\}\text{,}\) \(U^\bot\) is indeed a complement of \(U\) in the sense introduced in Theorem 1.8.10. It’s also fairly easy to see that \(\dim U + \dim U^\bot = n\) directly. If \(\ww\in U^\bot\text{,}\) and \(\{\uu_1,\ldots, \uu_k\}\) is a basis for \(U\text{,}\) then we have

\begin{equation*} \ww\dotp \uu_1= 0, \ldots, \ww\dotp \uu_k=0\text{,} \end{equation*}

and for an unknown \(\ww\text{,}\) this is simply a homogeneous system of \(k\) equations with \(n\) variables. Moreover, they are independent equations, since the \(\uu_i\) form a basis. We thus expect \(n-k\) free parameters in the general solution.

Theorem 3.3.9.

For any subspace \(U\) of \(\R^n\text{,}\) we have

\begin{equation*} U\oplus U^\bot = \R^n\text{.} \end{equation*}

Exercise 3.3.10.

Given subspaces \(U,W\) of \(\R^n\) with \(U\cap W=\{\zer\}\text{,}\) if \(U\oplus W=\R^n\text{,}\) then \(W=U^\bot\text{.}\)

True.
A subspace can have many complements, but only one orthgonal complement. For example, a complement to the \(x\) axis in \(\R^2\) is given by any other line through the origin, but only the \(y\) axis is orthogonal.
False.
A subspace can have many complements, but only one orthgonal complement. For example, a complement to the \(x\) axis in \(\R^2\) is given by any other line through the origin, but only the \(y\) axis is orthogonal.

Theorem 3.3.11.

Let \(U\) be a subspace of \(\R^n\text{,}\) with basis \(\{\uu_1,\ldots, \uu_k\}\text{.}\) Let \(A\) be the \(n\times k\) matrix whose columns are the basis vectors for \(U\text{.}\) Then \(U^\bot = \nll(A^T)\text{.}\)

Theorem 3.3.11 tells us that we can find a basis for \(U^\bot\) by solving the homogeneous system \(A^T\xx=\zer\text{.}\) Make sure you can see why this is true!

Example 3.3.12.

Let \(U = \{(a-b+3c, 2a+b, 3c, 4a-b+3c,a-4c)\,|\, a,b,c\in\R\}\subseteq \R^5\text{.}\) Determine a basis for \(U^\bot\text{.}\)

Solution.

First, we note that for a general element of \(U\text{,}\) we have

\begin{equation*} (a-b+3c, 2a+b, 3c, 4a-b+3c,a-4c) = a(1,2,0,4,1)+b(-1,1,0,-1,0)+c(3,0,3,3,-4)\text{,} \end{equation*}

so \(\{(1,2,0,4,1),(-1,1,0,-1,0),(3,0,3,3,-4)\}\) is a basis for \(U\text{.}\) (We have just shown that this set spans \(U\text{;}\) it is independent since the first two vectors are not parallel, and the third vector cannot be in the span of the first two, since its third entry is nonzero.) As in Theorem 3.3.11, we set \(A=\bbm 1\amp -1\amp 3\\2\amp 1\amp 0\\0\amp 0\amp 3\\4\amp -1\amp 3\\1\amp 0\amp -4\ebm\text{.}\)

To find a basis for \(U^\bot\text{,}\) we simply need to find the nullspace of \(A^T\text{,}\) which we do below.

Exercises Exercises

1.

Prove that for any subspace \(U\subseteq \R^n\text{,}\) \(\proj{U}{} + \proj{U^\bot}{}\) is the identity operator on \(\R^n\text{.}\)

Hint.

Given \(\xx\in\R^n\text{,}\) can you write it as a sum of an element of \(U\) and an element of \(U^\bot\text{?}\)

2.

Prove that for any subspace \(U\subseteq \R^n\text{,}\) \((U^\bot)^\bot = U\text{.}\)

Hint.

Show that \(U\subseteq (U^\bot)^\bot\text{,}\) and then use Remark 3.3.8 to show that the two spaces must have the same dimension.

3.

Let \(U\) and \(W\) be subspaces of \(\R^n\text{.}\) Prove that \((U+W)^\bot = U^\bot\cap W^\bot\text{.}\)

Hint.

One inclusion is easier than the other. Use Theorem 1.8.6 and Remark 3.3.8 to show that the dimensions must be equal.

4.

Given \(v=\begin{bmatrix}{{\frac{5}{3}}}\\ {{\frac{8}{3}}}\\ {{\frac{5}{3}}}\end{bmatrix}\text{,}\) find the coordinates for \(v\) in the subspace \(W\) spanned by \(u_{1}=\begin{bmatrix}6\\ 6\\ -1\end{bmatrix} \quad \text{ and } \quad u_{2}=\begin{bmatrix}2\\ -4\\ -12\end{bmatrix}\text{.}\)

Note that \(u_{1}\) and \(u_{2}\) are orthogonal.

\(v=\) \(u_{1}+\) \(u_{2}\)

5.

Let \(W\) be the set of all vectors \({\left[\begin{array}{c} x\cr y\cr x+y\cr \end{array}\right]}\) with \(x\) and \(y\) real. Find a vector whose span is \(W^{\perp}\text{.}\)

6.

Let \(\vec{u} = {\left[\begin{array}{c} -2\cr 7\cr 3\cr 1 \end{array}\right]}\) and \(\vec{v} = {\left[\begin{array}{c} -1\cr 4\cr 4\cr -3 \end{array}\right]}\text{,}\) and let \(W\) the subspace of \({\mathbb R}^4\) spanned by \(\vec{u}\) and \(\vec{v}\text{.}\) Find a basis of \(W^{\perp}\text{,}\) the orthogonal complement of \(W\) in \(\mathbb{R}^4\text{.}\)

7.

Let \(y=\begin{bmatrix}5\\ -3\\ -6\end{bmatrix}\text{,}\) \(u_{1}=\begin{bmatrix}-4\\ -6\\ -2\end{bmatrix}\text{,}\) and \(u_{2}=\begin{bmatrix}2\\ 2\\ -10\end{bmatrix}\text{.}\)

Compute the distance \(d\) from \(y\) to the plane in \({\mathbb R}^3\) spanned by \(u_{1}\) and \(u_{2}\text{.}\)

8.

Given \(\vec{v}={\left[\begin{array}{c} -7\cr -10\cr 2\cr 10 \end{array}\right]}\text{,}\) find the closest point to \(\vec{v}\) in the subspace \(W\) spanned by \({\left[\begin{array}{c} -3\cr -2\cr -4\cr 1 \end{array}\right]}\) and \({\left[\begin{array}{c} -2\cr -5\cr -1\cr -20 \end{array}\right]}\text{.}\)

9.

Find the orthogonal projection of \(\vec{v}={\left[\begin{array}{c} 9\cr 2\cr -19 \end{array}\right]}\) onto the subspace \(W\) of \({\mathbb R}^3\) spanned by \({\left[\begin{array}{c} -1\cr 6\cr -2 \end{array}\right]}\) and \({\left[\begin{array}{c} 6\cr -6\cr -21 \end{array}\right]}\text{.}\)

Prev Top Next