1 Principal Component Analysis | PCA | Dimensionality Reduction in Machine Learning by Mahesh Huddar

Summary

The video explains Principal Component Analysis (PCA) as a method for reducing the number of features in a dataset. The instructor works through a numerical example that reduces four data points with two features, (X_1) and (X_2), to a single new feature: the first principal component.

The example data points are ((4,11)), ((8,4)), ((13,5)), and ((7,14)). The feature means are (8) for (X_1) and (8.5) for (X_2).

PCA Method Demonstrated

1. Calculate the mean of each feature. Find the average value of (X_1) and (X_2). These means are used to center the data before calculating the principal components.

2. Calculate the covariance matrix. Calculate the covariance for each pair of features using the sample covariance formula, which divides by (n-1). The example produces:

[ \begin{bmatrix} 14 & -11\ -11 & 23 \end{bmatrix} ]

The diagonal entries represent each feature’s variance, while the off-diagonal entries represent the covariance between the features.

1. Find the eigenvalues. Solve the characteristic equation (\det(S-\lambda I)=0), where (S) is the covariance matrix. The eigenvalues are approximately (30.3849) and (6.6151).

2. Find and normalize the eigenvectors. For each eigenvalue, solve ((S-\lambda I)u=0) to obtain an eigenvector, then normalize it to unit length. The normalized eigenvector for the larger eigenvalue is approximately ((0.5574,-0.8303)). The other is approximately ((0.8303,0.5574)).

3. Choose the principal component. The eigenvector associated with the largest eigenvalue gives the first principal component—the direction that captures the greatest variance in the data. Since the example reduces two features to one, it keeps this component and discards the second.

4. Project the centered data onto the chosen eigenvector. Subtract the feature means from each observation, then take the dot product of each centered observation with the first eigenvector. This gives one principal-component score per example. The instructor illustrates the calculation for the first data point, obtaining a score of approximately (-4.30535). Using the rounded eigenvector values shown, the four scores are approximately (-4.305), (3.736), (5.693), and (-5.124).

Geometric Interpretation and Takeaway

PCA can be viewed geometrically: the eigenvectors define new directions through the data’s mean, and each data point is projected onto those directions. Keeping only the first direction represents the original two-feature dataset with one feature. If more features are retained, the next principal component comes from the next-largest eigenvalue.

Speakers or Sources Featured

• Mahesh Huddar — the video’s instructor and sole speaking voice.
• No other speakers or external sources are featured in the subtitles.