Video summary

W1_L3: Supervised learning: regression

Main summary

Key takeaways

Educational

Main ideas / concepts (Supervised Learning → Regression)

  • Supervised learning is framed as the most common paradigm in machine learning (often what people mean by “machine learning” unless otherwise specified).
  • Two key supervised-learning tasks:
    • Regression
    • Classification
  • This lecture focuses on regression, building from:
    • notation → learning goal → loss → a common model form.

Notation and definitions introduced

  • Real numbers: ( \mathbb{R} )
  • (d)-dimensional real vector space: ( \mathbb{R}^d )
    • Example vector in ( \mathbb{R}^3 ): ((3.6,\, 3.6,\, 5.2,\, -1.8))

Vector elements

  • A vector is often denoted by (x).
  • Coordinate access: (x_j) is the (j)-th coordinate of vector (x).

Vector norm / length

  • Denoted with a norm symbol; Euclidean length:

    • [ |x|^2 = x_1^2 + x_2^2 + x_3^2 + \cdots ]

    • [ |x| = \sqrt{x_1^2 + x_2^2 + x_3^2 + \cdots} ]

Indexing a dataset of vectors

  • Superscripts indicate different vectors in a collection.
  • Subscripts indicate coordinates within a vector.
  • Example: (x^{(i)}_j) = the (j)-th coordinate of the (i)-th vector.

Indicator variables

  • For a predicate that is true/false, the indicator value is:
    • 1 if true, 0 if false
  • Example:
    • Indicator of “2 is even” = 1
    • Indicator of “2 is odd” = 0

Core methodology of regression (what supervised learning “does”)

1) View supervised learning as curve fitting

  • Regression is described as curve fitting: given points, learn a function (“curve”) that fits them.

2) Training data structure

  • Data consists of:
    • Instances: (x^{(i)}), where each (x^{(i)} \in \mathbb{R}^d)
    • Labels/targets: (y^{(i)})
  • For regression:
    • Labels (y^{(i)}) are typically real-valued scalars.

3) Goal: learn a model/function (f)

  • Learn a function:
    • (f: \mathbb{R}^d \rightarrow \text{(label space)})
  • For regression specifically:
    • (f: \mathbb{R}^d \rightarrow \mathbb{R})
  • Requirement:
    • (f(x^{(i)})) should be close to (y^{(i)}).

4) Example: house price prediction

  • Predict house price using features like:
    • number of rooms
    • area
    • distance to metro
  • Each house is represented as a vector (x_i) (e.g., a 3D feature vector).
  • Each label (y_i) is the real-valued house price.

Loss function used to evaluate a regression model

  • A typical loss is squared error:
    • For a dataset point (i), error is:
      • (f(x^{(i)}) - y^{(i)})
    • Squared loss uses:
      • ((f(x^{(i)}) - y^{(i)})^2)
  • Properties emphasized:
    • Squared error is always non-negative
    • Loss = 0 only when:
      • (f(x^{(i)}) = y^{(i)}) for all points
  • The learning algorithm tries to find a model with low total loss (implicitly over the dataset).

Common model representation for regression: linear parameterization

Linear model form

  • [ f(x) = w^T x + b ]

  • Expanded:

    • [ f(x) = w_1 x_1 + w_2 x_2 + \cdots + w_d x_d + b ]

  • Parameters:

    • (w_1, \dots, w_d) and (b)
  • Interpretation (house example):
    • (w_1(\text{rooms}) + w_2(\text{area}) + w_3(\text{distance}) + b)

“How learning works” (illustrated as choosing the best model)

The lecture simplifies learning by showing selection among candidate models.

Simplified procedure shown

  • Choose candidate models (f) and (g) from a small set (for illustration).
  • For each model:
    • Compute predictions on training inputs: (f(x^{(i)})), (g(x^{(i)}))
    • Compute average squared loss (squared differences between predictions and true labels)
  • Select the model with smaller loss.

Illustrative example 1 (1D regression)

  • Inputs (x) are scalars represented as 1D vectors.
  • True labels (y) are given (e.g., points like (1 \to 2.1), (2 \to 3.9), etc.).
  • Two candidate models:
    • (f(x) = 2x)
    • (g(x) = x + 3)
  • Comparing losses shows (f) fits better, so:
    • lower loss ⇒ preferred model

Illustrative example 2 (3D regression / house-like features)

  • Inputs (x) are in (\mathbb{R}^3) (e.g., rooms, area, distance; the narration emphasizes rooms and distance).
  • Two candidate linear models:
    • (f = 2(\text{rooms}) - 0.5(\text{distance}))
    • (g = (\text{rooms}) + 2(\text{distance})) (plus a “distance”-type term)
  • Predictions are computed for each training point.
  • Loss for each model is computed via average squared error.
  • Again:
    • (f) yields smaller loss than (g), so (f) is selected.

Lesson / takeaway before moving on

  • Regression is essentially learning a function that fits labeled points by minimizing a squared-error-based loss.
  • The lecture suggests that in practice the algorithm:
    • chooses the best model from a potentially very large (“infinite” in principle) library
  • It concludes by transitioning to classification.

Speakers or sources featured

  • Lecture speaker (unnamed): the instructor presenting “machine learning foundations,” including the regression portion.

Original video