Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Mathematical Foundations for Data Science

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Part II — Calculus and Optimization

Topic 9 · Optimization and Gradient Descent

Print / Save as PDF
Lecture video — Topic 9: Optimization and Gradient Descent

Objectives and minima, stationary points, gradient descent with a fixed rate, worked examples and the calculus behind model training.

Flashcards · Optimization and gradient descent

10 words
WordMeaningExample
  • Optimization problem

    min L(x) over a feasible set D: choose x ∈ D to make the objective L as small as possible.

    Training: L is the loss, x the parameter vector.

  • Global vs local minimizer

    Global: L(x*) ≤ L(x) for all feasible x. Local: only for feasible x in a small ball around x*.

  • Stationary point

    An interior point with ∇L = 0. Every differentiable interior local minimiser is stationary, but not every stationary point is a minimum.

    x², −x², x³ are all stationary at 0: min, max, neither.

  • Convex objective

    For a differentiable convex L, ∇L(x) = 0 implies x is a global minimiser. Least-squares loss for an affine model is convex.

  • Gradient descent update

    x_{k+1} = x_k − η_k ∇L(x_k). Compute every component of the gradient at the same x_k before updating.

    x₀ = 0, η = 1/4, L = (x − 2)²: x₁ = 1, x₂ = 1.5.

  • Why minus the gradient

    L(x − ηg) = L(x) − η‖g‖² + o(η): a small step along −g strictly lowers L when g ≠ 0.

  • Learning rate and the error recurrence

    For L = (x − 2)², x_{k+1} − 2 = (1 − 2η)(x_k − 2). Convergence iff 0 < η < 1; η = 1 bounces, η > 1 diverges.

  • Stopping rules

    Test ‖g_k‖ ≤ ε for approximate stationarity. A small step η‖g‖ or small loss change can occur with a tiny η while the gradient is large.

  • Curvature and the step size

    For a quadratic with Hessian H: best step along −g is ‖g‖²/(gᵀHg); a fixed rate converges iff 0 < η < 2/λ_max.

    L = ½(u² + 9v²): η < 2/9.

  • Feature scaling

    Centering/rescaling a feature leaves the best-fitting line unchanged but reshapes the loss, changing the path and speed of gradient descent.

Practice in Python

Work through Section 4 of Lab 2 (Calculus and Optimization, MFDS_Lab2_Calculus_Optimization.ipynb), attached to the Part II materials lesson at the start of this part.