You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
Flashcards · Partial derivatives and the gradient
10 wordsF: ℝⁿ → ℝ assigns one real number to each input vector (x₁, …, xₙ).
∂F/∂xⱼ: the derivative along the slice where only xⱼ varies and all other variables are held fixed.
∇F: the column vector of all n partial derivatives, in input order.
F(a + h) ≈ F(a) + ∇F(a)·h for small h, if F is differentiable at a.
D_uF(a) = ∇F(a)·u: the rate of change per unit distance along a unit vector u (needs differentiability).
∇F points toward steepest increase, −∇F toward steepest decrease; the rates are ±‖∇F‖. A direction orthogonal to ∇F has first-order change 0.
All partial derivatives can exist (even be 0) while the function is not differentiable. Continuous partials near a point do guarantee differentiability.
With rᵢ = b + wxᵢ − yᵢ: ∇L_MSE = (2/m)(Σrᵢ, Σrᵢxᵢ), and ∇L_half = ½∇L_MSE. Vectorised: (1/m)Aᵀ(Aθ − y) for half-MSE.
d/dt F(x(t)) = ∇F(x(t))·x′(t): add each coordinate's partial derivative times its speed.
For ℓ = ½(p − y)² with p = σ(aᵀθ): ∇ℓ = (p − y)·p(1 − p)·a. The factor p(1 − p) is small when p is near 0 or 1.
Work through Section 3 of Lab 2 (Calculus and Optimization, MFDS_Lab2_Calculus_Optimization.ipynb), attached to the Part II materials lesson at the start of this part.