Лекция 2.1: Линейная регрессия.

Main ideas and lessons

• Why linear models (like linear regression) remain relevant

  • Simplicity and interpretability: learned weights can be inspected to see which input features matter and which do not.
  • Practical importance in high-stakes domains: e.g., credit scoring, where mistakes/outliers (approving a loan that will not be repaid) have serious financial consequences.
  • Strong assumptions reduce overfitting risk: linear models rely on simple structure/assumptions, making overfitting less likely (though it can still happen; remedies like regularization are mentioned later).
  • Ease of use: they typically require preprocessing (e.g., normalization) but otherwise work without special domain knowledge; predictions are produced relatively easily.
  • Data-efficiency: linear models can perform very well with limited data (contrasted with neural networks/trees that often need more data and behave differently).

• What linear regression is trying to do (problem setup)

  • Start with a dataset of records/samples.
  • Each sample contains:
    • n features (input variables, real numbers)
    • a target value (also real numbers)
  • Data representation:
    • Feature matrix (X): each row is an observation; each column is a feature
    • Target vector (y): a vertical vector of target values

• Loss function choice

  • Training uses a quadratic loss (squared error):
    • squared error is described as the most commonly used metric in regression
    • absolute/modulus error is mentioned as an alternative, but it is harder to optimize, so it’s used less
  • Framed as analogous to least squares (e.g., common in physics labs).

• Form of the linear regression model

  • Linear regression predicts the target as a weighted combination of features:
    • a bias/free term plus products of features with corresponding weights
  • Bias handling via augmentation:
    • add a constant 1 to the feature vector so the bias becomes part of the weight vector
  • Prediction can be written as a scalar product between weights and the (augmented) feature vector.

• How to increase model expressiveness

  • Linear regression is called “linear” because it is linear in parameters (weights), not necessarily linear in inputs.
  • Enrich the feature set by creating new features:
    • use original features
    • apply transformations (examples: logarithm, double logarithm)
    • use powers of original features
    • include other nonlinear derived numeric features (example mentioned: “Manhattan …” leading to multiple real-valued outputs, described as still usable)
    • include pairwise products/interactions between different features
  • Motivation:
    • Without interaction terms, a model cannot represent certain combined effects (e.g., when both (x_1) and (x_2) being simultaneously large/small changes the target in ways a plain linear model can’t capture).
  • Geometric intuition:
    • with few features, the model corresponds to a hyperplane
    • with many engineered features, it can become a more complex surface in the original feature space

---

Methodology / instructions (how the solution is derived)

A) Define the linear regression model (with matrix notation)

• Represent data as:
  • (X): an (m \times n) matrix ( (m) observations/samples, (n) features after augmentation/engineering)
  • (y): an (m \times 1) vector of target values
• Choose weights:
  • (\omega): an (n \times 1) weight vector
• Prediction:
  • (\hat{y} = X\omega)

B) Choose the loss function (quadratic loss / least squares)

• Error vector:
  • (e = \hat{y} - y = X\omega - y)
• Objective:
  • minimize the sum of squared errors:
    • (L(\omega) = |X\omega - y|^2)

C) Derive the optimal weights

• Take the derivative of the loss w.r.t. (\omega) and set it to zero.
• The resulting closed-form solution is presented (conceptually):
  • (\omega = (X^T X)^{-1} X^T y)
• Preconditions mentioned:
  • the formula involves an inverse, so it requires linear independence of columns of (X)
  • if columns are linearly dependent, the inverse may not exist

D) Handle different matrix “shape” cases

• Case 1: (n) features equal to number of observations (square-ish / solvable linear system)
  • the lecture references solving a regular linear system
  • using the inverse of (X) is possible in ideal circumstances

• Case 2: more observations than features (common case)

  • the system is generally overdetermined (no exact solution satisfies all equations perfectly)
  • use the pseudo-inverse concept:
    • for “tall and thin” matrices, the pseudo-inverse is described as based on (X^T X) and inversion
    • it yields weights that give the smallest squared error

• Failure modes / numerical issues

  • If (X) has linearly dependent columns:
    • pseudo-inverse/inverse assumptions break down or require caution
  • Even if ((X^T X)^{-1}) exists, computing an inverse can be:
    • computationally expensive for very large datasets
    • numerically unstable when columns are approximately linearly dependent
    • prone to large inverse values that may not fit well in memory/precision

---

Speakers / sources featured

• Alena (explicitly referenced: “let’s move on to Alena and regression”)
• No other named speakers, authors, or external sources are mentioned in the subtitles.