Topic 8 · Partial Derivatives and the Gradient
[video] Lecture video — Topic 8: Partial Derivatives and the Gradient — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794892661691-ce8b6a488aa1a892.mp4
Flashcards · Partial derivatives and the gradient
Function of several variables — F: ℝⁿ → ℝ assigns one real number to each input vector (x₁, …, xₙ).
F(b, w) = b² + 3bw + 2w².
Partial derivative — ∂F/∂xⱼ: the derivative along the slice where only xⱼ varies and all other variables are held fixed.
∂F/∂b = 2b + 3w; at (1, 2) it equals 8.
Gradient — ∇F: the column vector of all n partial derivatives, in input order.
∇F(1, 2) = (8, 11)ᵀ.
Linear approximation — F(a + h) ≈ F(a) + ∇F(a)·h for small h, if F is differentiable at a.
Δx₁ = 0.01, Δx₃ = −0.02 gives ΔF ≈ 4(0.01) + 1(−0.02) = 0.02.
Directional derivative — D_uF(a) = ∇F(a)·u: the rate of change per unit distance along a unit vector u (needs differentiability).
u = (3/5, 4/5): D_uF = 68/5.
Steepest direction — ∇F points toward steepest increase, −∇F toward steepest decrease; the rates are ±‖∇F‖. A direction orthogonal to ∇F has first-order change 0.
Partials do not imply differentiability — All partial derivatives can exist (even be 0) while the function is not differentiable. Continuous partials near a point do guarantee differentiability.
G = s²t/(s² + t²): ∇G(0, 0) = 0 but D(1,1)G(0, 0) = 1/2.
Gradient of the squared-error loss — With rᵢ = b + wxᵢ − yᵢ: ∇L_MSE = (2/m)(Σrᵢ, Σrᵢxᵢ), and ∇L_half = ½∇L_MSE. Vectorised: (1/m)Aᵀ(Aθ − y) for half-MSE.
Chain rule along a path — d/dt F(x(t)) = ∇F(x(t))·x′(t): add each coordinate's partial derivative times its speed.
F(1 + t, 2 − t) = 15 − 3t, rate −3.
Gradient of a sigmoid loss — For ℓ = ½(p − y)² with p = σ(aᵀθ): ∇ℓ = (p − y)·p(1 − p)·a. The factor p(1 − p) is small when p is near 0 or 1.
[markdown] content available only on the platform
