Derivatives, gradient, and Hessian

For a differentiable function , the gradient is the vector of partial derivatives:

The Hessian is the matrix of second derivatives:

In one dimension, the derivative generalizes the instantaneous rate of change; the gradient and Hessian extend this idea to multivariate functions.

Intuition

Derivative / gradient: local direction and rate of change. points toward the steepest increase of ; indicates the steepest descent.
Hessian: local curvature. Its eigenvalues describe whether the function is more concave or convex along each direction.
Linear approximation: near ,

Quadratic approximation: adding curvature,

This is the foundation of gradient descent, Newton's method, and Taylor series.

Notation and dimensions

Object	Dimension	Role
Partial derivative	Scalar	Change in when only moves
Gradient	Vector	Direction of steepest ascent
Jacobian	Matrix	Gradient of a vector-valued function
Hessian	Matrix	Local curvature; symmetric if is
Directional derivative	Scalar	for a unit vector

Calculation rules

Rule	Gradient	Comment
Linearity		Basis of all symbolic computation
Product		Useful for composed losses
Chain rule		Foundation of backpropagation in deep learning
Quadratic form	if is symmetric	Appears in OLS, ridge, and Markowitz
Hessian symmetry		Under regularity; simplifies matrix computations

Examples

Function	Gradient	Hessian



(softplus)
Least squares

Link with convexity

For a twice-differentiable function:

(positive definite) ⟹ is strictly convex locally;
everywhere ⟹ is convex;
a point where and is a local minimum; on a convex domain, it is the global minimum.

See Convexity for the full characterizations (Jensen, gradient criterion).

Applications

Optimization — Gradient descent: update — see Gradient descent. Newton's method: uses to correct direction and converge faster near the minimum. Optimality conditions: without constraints, an interior optimum satisfies .

Statistics and ML — Linear regression: setting the quadratic loss gradient to zero yields the OLS estimator. L2 regularization: adds to the Hessian, improving conditioning. Backpropagation: gradient propagation via the chain rule on a computation graph.

Finance — Greeks (delta, gamma) are derivatives of a derivative price with respect to risk factors — the same differential logic applied to pricing.

Limits and pitfalls

Situation	Issue	Consequence
Non-differentiability	E.g. at 0	Classical gradient does not exist; use subgradients
Ill-conditioned Hessian	Very spread eigenvalues	Unstable Newton; slow gradient descent in some directions
Saddle point	but indefinite Hessian	Critical point that is neither min nor max; trap for naive Newton
Finite differences	Noisy numerical approximation	Rounding errors; prefer AD (autodiff) when possible
High dimension	Storing costs	Quasi-Newton (BFGS, L-BFGS) approximates curvature without an explicit Hessian