Lecture 16: Geometry of Singular Value Decomposition | SVD | GATE DA | Linear Algebra Sachin Mittal

Summary

The lecture explains the geometric meaning of the singular value decomposition (SVD), then applies SVD facts to practice questions.

1. SVD and the four fundamental subspaces

For a matrix (A = U\Sigma V^T), the singular vectors describe the fundamental subspaces:

• The left singular vectors (u_1,\ldots,u_r) corresponding to nonzero singular values form a basis for the column space of (A).
• The right singular vectors (v_{r+1},\ldots,v_n), corresponding to zero singular values, form a basis for the null space of (A).
• The right singular vectors associated with nonzero singular values span the row space.
• The remaining left singular vectors span the left null space.

These relationships follow from (Av_i = \sigma_i u_i): a zero singular value sends its right singular vector to zero, while a nonzero singular value maps it to a column-space direction.

2. Geometric interpretation: rotate, scale, rotate

Applying (A) to a vector can be understood as three successive transformations:

1. Apply (V^T): Change the vector’s orientation or coordinates.
2. Apply (\Sigma): Scale along coordinate directions by the singular values. A circle or sphere can become an ellipse or ellipsoid.
3. Apply (U): Rotate the result again.

Thus, (U) and (V^T) change orientation without changing lengths, while (\Sigma) determines the stretching or shrinking. If a singular value is zero, its direction is collapsed.

3. Properties of orthonormal matrices

For an orthonormal matrix (U), (U^TU = I), equivalently (U^{-1} = U^T). This has two important geometric consequences:

• Length preservation: (|Ux| = |x|). The proof is [ |Ux|^2 = (Ux)^T(Ux) = x^TU^TUx = x^Tx = |x|^2. ]

• Angle preservation: Applying the same orthonormal matrix to two vectors preserves their dot product and lengths, and therefore preserves the angle between them.

An orthonormal matrix acts as a rotation or reflection rather than stretching vectors. Its determinant is (+1) or (-1).

4. How singular values control lengths

Since the orthonormal factors preserve length, the singular values determine how much a vector can be stretched or shrunk. For a vector (x), the standard bounds are

[ \sigma_{\min}|x| \leq |Ax| \leq \sigma_{\max}|x|. ]

For a unit vector, the output length lies between the smallest and largest singular values. The lecturer uses this to reject a proposed result whose length falls outside the range allowed by the singular values.

5. SVD facts used in the practice questions

The lecture works through questions involving these rules:

• The singular values of (A) are the square roots of the eigenvalues of (A^TA).
• The number of nonzero singular values equals (\operatorname{rank}(A)).
• If (A) is orthogonal, all its singular values are (1).
• For a square matrix, (|\det(A)|) is the product of its singular values. The sign of (\det(A)) depends on the orthonormal factors.
• Singular values are not generally the absolute values of the eigenvalues of (A), and the number of nonzero eigenvalues of (A) need not equal the number of nonzero singular values.
• If (A) is invertible, the singular values of (A^{-1}) are the reciprocals of those of (A).
• Multiplying a matrix by (-1) does not change its singular values; the sign can be absorbed into an orthonormal factor.
• For a symmetric matrix with eigenvalues (3) and (-2), the corresponding singular values are (3) and (2). Sign choices in singular vectors can vary.

The instructor also discusses reading null-space and column-space bases directly from an SVD, and finding singular vectors and values for a symmetric-matrix example.

6. Practice and class plan

The instructor begins a set of 30 SVD questions. Several are solved during the lecture, including questions on subspaces, eigenvalues and singular values, determinants, inverses, and orthogonal matrices. The remaining questions are assigned for students to try before being discussed in the next class. A quiz based on SVD is also announced for the following day.

Speakers and sources featured

• Sachin Mittal — Primary lecturer and presenter.
• Students participating through the chat — Brief answers and questions are acknowledged. Names audible in the subtitles include Yash, Aay, Malali, and Bachu Paul, though some spellings may be inaccurate.
• Harvard University — Cited by the lecturer as the source of the length- and angle-preservation properties.
• MIT — Cited as the source of one practice question.
• An unspecified university — Cited as the source of another practice question.