Back to the lessonPreparing the printout…

Mathematical Foundations for Data Science · Part II — Calculus and Optimization

Topic 9 · Optimization and Gradient Descent

[video] Lecture video — Topic 9: Optimization and Gradient Descent — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794892663053-f3c743be954cb816.mp4

Flashcards · Optimization and gradient descent

Optimization problem — min L(x) over a feasible set D: choose x ∈ D to make the objective L as small as possible.

Training: L is the loss, x the parameter vector.

Global vs local minimizer — Global: L(x*) ≤ L(x) for all feasible x. Local: only for feasible x in a small ball around x*.

Stationary point — An interior point with ∇L = 0. Every differentiable interior local minimiser is stationary, but not every stationary point is a minimum.

x², −x², x³ are all stationary at 0: min, max, neither.

Convex objective — For a differentiable convex L, ∇L(x) = 0 implies x is a global minimiser. Least-squares loss for an affine model is convex.

Gradient descent update — x_{k+1} = x_k − η_k ∇L(x_k). Compute every component of the gradient at the same x_k before updating.

x₀ = 0, η = 1/4, L = (x − 2)²: x₁ = 1, x₂ = 1.5.

Why minus the gradient — L(x − ηg) = L(x) − η‖g‖² + o(η): a small step along −g strictly lowers L when g ≠ 0.

Learning rate and the error recurrence — For L = (x − 2)², x_{k+1} − 2 = (1 − 2η)(x_k − 2). Convergence iff 0 < η < 1; η = 1 bounces, η > 1 diverges.

Stopping rules — Test ‖g_k‖ ≤ ε for approximate stationarity. A small step η‖g‖ or small loss change can occur with a tiny η while the gradient is large.

Curvature and the step size — For a quadratic with Hessian H: best step along −g is ‖g‖²/(gᵀHg); a fixed rate converges iff 0 < η < 2/λ_max.

L = ½(u² + 9v²): η < 2/9.

Feature scaling — Centering/rescaling a feature leaves the best-fitting line unchanged but reshapes the loss, changing the path and speed of gradient descent.

[markdown] content available only on the platform

Mathematical Foundations for Data Science · Topic 9 · Optimization and Gradient Descent · printed on November 20, 2026