Video summary

Lecture 2 | Image Classification

Main summary

Key takeaways

Educational

Main Ideas & Lessons (Lecture 2: Image Classification / CS231N)

1) Lecture purpose and framing

  • Lecture 2 shifts from the “big picture” of computer vision (Lecture 1) into the mechanics of how learning algorithms work.
  • Today’s focus:
    • k-nearest neighbors (k-NN) as a starting point
    • moving toward linear classification, a foundation for neural networks and CNNs

2) Administrative / course logistics

  • Piazza is the primary communication channel.
    • Many students are not yet signed up.
    • Questions about projects, midterm attendance, poster attendance, etc. should go to Piazza for faster TA responses (course emails can get lost).
  • SCPD Piazza access issue
    • SCPD students should receive a @stanford.edu email to sign into Piazza.
  • Assignment 1
    • Will be posted later today (likely this afternoon).
    • A “last year” version is available now and is similar in content.
    • Updates:
      • switching to Python 3 instead of Python 2.7
      • minor cosmetic changes
    • Assignment components:
      • implement k-NN classifier (covered in this lecture)
      • implement several linear classifiers (including SVM and Softmax)
      • implement a simple two-layer neural network
    • These topics continue across subsequent lectures.
  • Python + NumPy requirement
    • The class uses Python and NumPy heavily.
    • Emphasis: vectorized operations are crucial for efficient code.
    • Students should start early and read the NumPy/Python tutorial on the course website.
  • Google Cloud support
    • The course is supported via Google Cloud (similar to AWS).
    • Plan: students can run assignments on provided cloud instances, including GPU support.
    • Students will receive free Google Cloud credit coupons.
    • More details will be posted on Piazza.

3) Why image classification is hard for machines

  • Image classification task:
    • Input: an image
    • Output: one of a fixed set of category labels (e.g., cat/dog/truck)
  • Humans succeed, but machines face the semantic gap:
    • Humans perceive “cat-ness” holistically.
    • Computers see raw pixel grids (e.g., 800×600 pixels, each pixel has RGB values → huge arrays of numbers).
    • There’s a disconnect between semantic concepts (labels) and pixel-level representations.
  • Algorithms must handle variations:
    • Viewpoint changes (camera moves, pixels change)
    • Illumination changes (lighting conditions)
    • Deformation / pose changes (cats appear in many poses)
    • Occlusion (only part visible)
    • Background clutter (foreground similar to background)
    • Intraclass variation (same category can look different)

4) Data-driven approach: the key modern insight

  • Instead of manually writing brittle, explicit rules (e.g., “ears/eyes/mouth edges → cat rules”):
    • rule-based systems don’t scale to new categories
    • they are brittle and require rewriting for each object type
  • Data-driven learning
    • Collect large labeled datasets (e.g., via Google Image Search or existing datasets).
    • Train a model to learn patterns mapping pixels/features → labels.
    • Use the trained model to predict on new images.
  • API shift:
    • From: a single classify(image) function
    • To:
      • train(images, labels) → model
      • predict(model, images) → predictions

Method 1: k-Nearest Neighbors (k-NN)

5) Simple baseline: nearest neighbor

Core idea

  • Training step: memorization
    • No learning beyond storing the dataset.
  • Prediction step:
    • For a new image, find the most similar training image
    • Predict that closest example’s label

Dataset example: CIFAR-10

  • 10 classes (e.g., airplane, automobile, bird, cat, etc.)
  • ~50,000 training images per class (roughly evenly distributed)
  • 10,000 test images
  • Nearest neighbors often look similar visually, but can still misclassify.

6) Distance metric: comparing images

To compare two images, choose a distance function:

  • L1 distance (Manhattan distance)
    • For each pixel: absolute difference
    • Sum across all pixels
  • Example computation:
    • compute per-pixel differences between a test and training image
    • sum to produce a single distance value (e.g., “456” in the lecture)
  • Implementation note
    • NumPy vectorization enables short, efficient code:
      • “training” = memorize
      • “testing” = compute distances to all training examples and pick the closest

7) Practical complexity tradeoff (training vs testing)

  • Nearest neighbor:
    • Training: fast/constant time (store data)
    • Testing: slow
      • compare each test image to all N training examples
  • This is the opposite of what many deployed systems want:
    • often: heavier compute during training, faster inference at test time
  • The lecture foreshadows CNNs/parametric models as a way to reverse this tradeoff.

8) Decision regions & limitations of k-NN with K=1

  • In a 2D toy visualization:
    • colors show the predicted class per region based on nearest training point
  • Observed problems:
    • a single noisy/spurious point can create small incorrect “islands”
    • boundaries become jagged and unstable

Detailed instruction list: k-NN classifier (major steps)

  1. Choose hyperparameters
    • K (number of neighbors)
    • a distance metric (e.g., L1 or L2)
  2. train(X_train, y_train)
    • store training data and labels (no parameter learning)
  3. predict(X_test)
    • for each test image:
      • compute distances to all training images using the chosen metric
      • select the K nearest training images
      • do majority voting among their labels
      • if no clear majority:
        • in the demo, classify as white/unassigned
  4. Output
    • predicted class labels for test images

9) k-NN generalization: choose K > 1

  • Uses K nearest neighbors instead of just the closest one.
  • Majority voting:
    • smooths decision boundaries
    • reduces sensitivity to single noisy points
  • Effects described:
    • K=1: jagged boundaries and noisy islands
    • K=3: islands disappear, boundaries smoother
    • K=5: smoother/nicer boundaries

10) L1 vs L2 distance (and geometry implications)

  • L2 (Euclidean) distance
    • uses the square root of the sum of squared differences (distance = norm)
  • Key geometric difference:
    • L1 depends on coordinate orientation (rotating axes changes L1 results)
    • L2 is rotation-invariant
  • Implication:
    • distance metrics affect the shape of decision regions
    • L1 boundaries tend to align with axes; L2 yields more “circular” boundaries

11) Hyperparameters: K and distance metric

  • Hyperparameters are not learned directly from training data; they’re chosen ahead of time.
  • How to choose them:
    • evaluate multiple choices and select the best based on validation performance
  • Common mistakes to avoid:
    • Don’t select hyperparameters using training-set accuracy
      • e.g., K=1 may perfectly classify training data but generalize poorly
    • Don’t use the test set to tune hyperparameters
      • leads to overly optimistic, unrepresentative results on unseen data

12) Correct evaluation protocol

  • Use three splits:
    • training set: fit/train (for k-NN: store data)
    • validation set: choose hyperparameters (evaluate models)
    • test set: final evaluation only once (reported in papers/reports)
  • Strict separation:
    • touch the test set only at the end to avoid dishonest/overfit reporting

13) Cross-validation (especially for smaller data)

  • If data is small, use K-fold cross-validation:
    • keep test set only for final evaluation
    • split remaining data into folds
    • rotate the validation fold and average performance
  • Example: 5-fold CV
    • train on 4 folds, validate on the 5th; repeat for each fold
  • Benefits:
    • more confidence in results and estimates variability
  • Mentioned:
    • less common in deep learning due to computational cost

14) Why k-NN is rarely used for images in practice

Even with cross-validation and tuning, k-NN struggles because:

  • Slow inference at test time
  • Distance metric mismatch with perception
    • transformations may appear differently to humans but be similarly distant under pixel-level metrics
    • example idea: blocking/tinting/shifting can yield similar L2 distances despite different perceived images
  • Curse of dimensionality
    • nearest neighbors must be dense in feature space
    • density requirement grows exponentially with dimensionality
    • would require an infeasible number of training examples to cover pixel space well

Method 2: Linear Classification (Next Major Topic)

15) Motivation and role in deep learning

  • Linear classifiers are introduced as a foundational building block.
  • Neural networks are described as “Lego blocks”:
    • linear classification is a basic component later architectures build on
  • Preview of modularity:
    • Example system (image captioning):
      • convolutional network for image features
      • recurrent network for language
      • combined like Lego-like modules

16) Parametric model vs non-parametric k-NN

  • Linear classifier is a parametric model:
    • has learnable weights/parameters rather than storing all training examples for inference
  • For CIFAR-10:
    • input image: 32×32×3 = 3072 values
    • output: scores for 10 classes

17) Linear classifier functional form

  • Represent input image as a vector X (length 3072).
  • Learn weight matrix W with shape 10 × 3072.
  • Compute class scores:
    • F(X, W) = W · X
  • Often include a bias term (a length-10 vector):
    • provides class-specific offsets independent of the input
    • especially helpful for unbalanced datasets

18) Interpretation of the linear classifier

  • Template matching viewpoint:
    • each row of W can be visualized as a “template” for a class
    • dot product between template and image vector ≈ similarity score
  • Geometric viewpoint:
    • in high-dimensional space, the model forms linear decision boundaries (hyperplanes)
    • training adjusts boundary placement to separate categories

19) Limitations of linear classifiers

  • One template per class → averages across variations.
    • examples described:
      • “plane template” may look like a generic blue blob
      • “car template” may not look like a real car (it averages diverse appearances)
      • “horse template” may reflect typical background (green grass) and produce odd artifacts (e.g., two heads)
  • Linear boundaries can fail for non-linearly separable patterns.
    • example: parity/odd-even-like problems
  • Struggles with multimodal data
    • if a class appears as separate “islands” in feature space, one linear boundary may not separate it cleanly

20) What remains for next lecture

  • The functional form is established (how scores come from W and X).
  • Next lecture will cover:
    • how to choose/learn the correct weights W
    • connected ideas:
      • loss functions
      • optimization
      • leading toward neural networks and convolutional networks (ConvNets)

Speakers / Sources Featured

  • Course instructor (speaker at lectern) — primary speaker; name not provided in subtitles
  • Hubel and Wiesel — referenced regarding the importance of edges in visual recognition

Original video