Video summary

Lec 08 Varients of Gradient descent and Momentum based techniques

Main summary

Key takeaways

Educational

Main ideas & lessons from the lecture

1) Variants of Gradient Descent

Core gradient descent update rule

Parameters (weights/biases) are updated iteratively:

[ \theta_{\text{new}} = \theta_{\text{old}} - \eta \cdot \nabla_{\theta} L ]

Where:

  • (\eta) = learning rate
  • (\nabla_{\theta} L) = gradient of the loss (cost) with respect to the parameters

Batch Gradient Descent

  • What happens per iteration
    • Uses all data points to compute the gradient.
    • Updates weights/biases after averaging over the entire dataset.
  • Behavior
    • Stable convergence
    • Slow on large datasets (more computation per update)

Stochastic Gradient Descent (SGD)

  • What happens per iteration
    • Uses one randomly selected data point per iteration.
    • Updates weights/biases based on that sample’s gradient.
  • Behavior
    • Faster convergence
    • Noisier / oscillatory trajectory
  • Why noisy?
    • With one sample, you get a point estimate of the true gradient rather than the true gradient (which would require all data).

Mini-batch Gradient Descent

  • What happens per iteration
    • Uses a small subset of data per update.
  • Behavior
    • Positioned between batch and SGD:
      • Less noisy than SGD
      • Typically faster than full batch gradient descent
      • Takes fewer iterations than batch, but usually more than pure SGD

2) Why contours are used (visualization concept)

The lecture explains gradient descent using contour plots of a loss surface.

  • A 3D loss surface (weights, biases, cost/error) is hard to visualize directly.
  • Contours represent locations with equal cost.
  • Closely spaced contours
    • imply a steep slope
    • changes in cost are faster
  • Widely spaced contours
    • imply a gentle slope/plateau
    • progress is slower

3) Problem with plain gradient descent: plateau regions

Gradient descent can waste time in flat/low-slope (plateau) regions because:

  • small gradient → small weight updates
  • leads to slow navigation toward minima

4) Momentum-based optimization

Intuition

Like a ball rolling downhill:

  • Momentum helps it move through flat regions
  • Helps reduce issues from noisy gradients and oscillations

Update rule (as presented)

  • Momentum term accumulates previous updates/history:

[ update_t = \gamma \cdot update_{t-1} + \eta \cdot g_t ] [ W_{t+1} = W_t - update_t ]

Where:

  • (\gamma) = momentum hyperparameter (between 0 and 1)
  • (g_t) = gradient at time (t)

Effect

  • Oscillations are reduced / smoother

Trade-off

  • Momentum can still overshoot the minimum, causing oscillations near convergence (some wasted iterations before settling)

5) Nesterov Accelerated Gradient (NAG)

Core idea (look-ahead)

“Look ahead before you jump.” Before committing, estimate where momentum will take you.

Motivation

  • Reduce overshoot oscillations caused by standard momentum.

Update concept (as described)

  • Compute a look-ahead position using the previous update:

[ \text{look-ahead weights} = W_t - \gamma \cdot update_{t-1} ]

  • Then compute the gradient at that look-ahead point and form the momentum update:

[ update_t = \gamma \cdot update_{t-1} + \eta \cdot \nabla L(\text{look-ahead}) ]

  • Finally:

[ W_{t+1} = W_t - update_t ]

  • Same concept extends to biases (mentioned).

6) Challenge of a single learning rate & learning rate schedules

Gradients vary across the loss landscape:

  • steep slope regions → large gradient magnitude
  • gentle slope / plateau → small gradient magnitude

Why one learning rate (\eta) may fail:

  • If (\eta) is too large:
    • in steep regions, updates become drastic → instability/overshooting
  • If (\eta) is too small:
    • in flat regions, updates become tiny → slow progress

Learning rate decay methods (as given)

  • Time-based decay [ \eta_t = \eta_0 \cdot \frac{1}{1 + kT} ] As (T) increases, learning rate decreases.

  • Exponential decay [ \eta_t = \eta_0 \cdot e^{-k t} ] As time increases, learning rate decreases exponentially.

7) Adaptive learning rate and handling sparse vs dense features

Sparse dataset concept

Example: one-hot encoded movie watch/rating data.

  • Many inputs are 0, only a few are 1
  • For sparse features:
    • gradients for associated weights are often small
    • updates happen slowly / infrequently

Dense dataset concept

  • Inputs are non-zero across most features
  • gradients and updates tend to be larger consistently

8) AdaGrad

Core idea

Adjust learning rate based on history of gradients/updates:

  • past gradients large → learning rate should decrease
  • past gradients small → learning rate should not decrease as much

Update rules (as presented)

  • Accumulator: [ v_t = v_{t-1} + g_t^2 ]

  • Parameter update: [ W_{t+1} = W_t - \frac{\eta}{\sqrt{v_t} + \epsilon} \cdot g_t ]

Takeaways

  • Slow decay for sparse data (updates/gradients accumulate slowly)
  • Faster/aggressive decay for dense features (since (v_t) grows quickly)

Limitation

AdaGrad can reduce learning rate too aggressively, potentially causing it to get stuck near convergence (learning rate near zero too early).

9) RMSProp

Core idea

Use an exponentially decaying average of squared gradients instead of a cumulative sum.

  • Prevents learning-rate from decaying to near-zero too quickly (fixes AdaGrad’s aggressive decay)

Update concept

[ v_t = \beta v_{t-1} + (1-\beta) g_t^2 ]

  • (\beta) is typically 0.9 or 0.95

[ W_{t+1} = W_t - \frac{\eta}{\sqrt{v_t} + \epsilon} \cdot g_t ]

Effect

  • Exponentially decayed history means older large gradients matter less
  • learning rate doesn’t collapse as rapidly as AdaGrad

10) Adam (Adaptive Moment Estimation)

Core idea

Combine:

  • Momentum (first moment / gradient history)
  • RMSProp-like adaptive learning rate (second moment / squared gradients)

As presented (conceptual forms)

  • Momentum-like term:
    • (m_t) updated using (\beta_1), then bias-corrected to (\hat{m}_t)
  • Second moment term:
    • (v_t) updated using (\beta_2), then bias-corrected to (\hat{v}_t)

Update resembles:

  • divide the step by (\sqrt{\hat{v}_t} + \epsilon)
  • multiply by (\hat{m}_t)

Practical claim

Adam is commonly a default choice in real-world applications (though the “best” optimizer can depend on the task).

11) Adadelta

Core idea

Avoid using an explicit initial learning rate hyperparameter.

  • Dynamically tunes step sizes using running averages

Update logic (as presented)

  • Weight update: [ W_{t+1} = W_t + \Delta W_t ]

  • (\Delta W_t) uses a ratio of:

    • accumulated gradient information in the denominator
    • accumulated past updates in the numerator (one step behind)
  • Uses exponentially decaying running averages:
    • (v_t) from squared gradients
    • (u_t) (or similar) from squared parameter updates

Why it works (as explained)

  • Effective learning rate becomes context-dependent:
    • steep slope → denominator grows → smaller effective step
    • gentle slope / plateau → numerator relatively dominates → larger effective step
  • Reduces the need for tuning an initial (\eta)

12) Final recap of what was learned

  • Gradient descent variants:
    • Batch: stable but slow
    • SGD: fast but noisy
    • Mini-batch: compromise, commonly used
  • Momentum reduces plateau issues but may overshoot and oscillate near minima.
  • Nesterov reduces oscillations by looking ahead before updating.
  • Learning rate schedules address issues with fixed (\eta).
  • Adaptive methods:
    • AdaGrad: good for sparse but can decay too aggressively
    • RMSProp: fixes AdaGrad’s aggressive decay using exponential averaging
    • Adam: combines momentum + RMSProp benefits
    • Adadelta: removes need for initial learning rate via dynamic tuning
  • Next lecture planned: regularization techniques.

Speakers / sources featured

  • Speaker/Instructor: Not explicitly named (lecturer delivering “Lec 08” content)
  • External sources: None explicitly cited

Original video