Partial Differentiation

Consider a scalar field of the form:

This is a function that takes a point in $n$-dimensional space and returns a single real number — like a temperature field in a room, or an elevation map over terrain.

Directional Derivative

Recall from 1D calculus that the derivative of $f$ at a point $a$ is:

This is the slope of the function at $a$: how much does $f$ change when we take a tiny step of size $h$ away from $a$? Geometrically, it is the same intuition behind $\tan(\alpha) = \frac{\sin(\alpha)}{\cos(\alpha)}$ — the slope of a line is simply the ratio of the vertical rise to the horizontal run, i.e., $\frac{\Delta y}{\Delta x} = \frac{f(a+h) - f(a)}{h}$.

In multiple dimensions, the same question becomes richer: we can step away from a point in infinitely many directions. This is where the directional derivative comes in — it tells us the rate of change of $f$ when we stand at a point $\mathbf{a}$ and look in a specific direction $\mathbf{v}$.

Here, $\mathbf{a} \in D$ is the base point — the location in the domain where we want to measure how fast $f$ is changing. Think of $\mathbf{a}$ as your exact current GPS coordinate while standing perfectly still. $\mathbf{v}$ is the direction you are facing. Crucially, the directional derivative is a local quantity — it captures the behavior of $f$ specifically at $\mathbf{a}$, not globally.

For each unit vector $\mathbf{v}$ (with $\|\mathbf{v}\| = 1$), we take a tiny step of size $h$ from $\mathbf{a}$ in the direction $\mathbf{v}$, measure the change in $f$, and divide by the step size — the same slope formula as before, now generalized to any direction in $n$-dimensional space:

As $h \to 0$, the step size becomes infinitesimally small, and we recover the instantaneous rate of change in the chosen direction.

All three notations — $\frac{\partial f}{\partial \mathbf{v}}(\mathbf{a})$, $\partial_\mathbf{v} f(\mathbf{a})$, and $f_\mathbf{v}(\mathbf{a})$ — refer to the same quantity: the rate of change of $f$ at the base point $\mathbf{a}$ along the direction $\mathbf{v}$.

Requiring $\mathbf{v}$ to be a unit vector ensures that the step $h\mathbf{v}$ has length exactly $|h|$, making the derivative a pure measure of directional rate of change, independent of the magnitude of $\mathbf{v}$.

Since $\mathbf{v}$ can point in any direction on the unit sphere in $\mathbb{R}^n$, there are infinitely many directional derivatives at any given point $\mathbf{a}$. In principle, this limit may not exist for every combination of $f$, $\mathbf{a}$, and $\mathbf{v}$. However, throughout this course we will work with sufficiently smooth functions and always assume the limit exists — the “happy scenario.”

Partial Derivatives

Having infinitely many directions to check is impractical. The saving insight is that we are working in a linear vector space: any direction $\mathbf{v}$ can be expressed as a linear combination of the $n$ vectors of the standard basis (also called the canonical basis) of $\mathbb{R}^n$. For smooth functions, this means the behavior of $f$ in every direction is fully determined by its behavior along the $n$ coordinate directions — one per argument of the function. We do not need to sweep through all directions; the $n$ coordinate directions are enough.

Substituting $\mathbf{v} = \mathbf{e}_i$ into the directional derivative formula means we step from $\mathbf{a}$ by a tiny amount $h$ purely along the $i$-th axis, holding all other coordinates fixed. The resulting limit is the rate of change of $f$ with respect to its $i$-th argument $x_i$.

As always, we assume this limit exists. Each partial derivative answers a single focused question: how fast does $f$ change when only the $i$-th coordinate of $\mathbf{a}$ is nudged, while all others stay fixed? Together, the $n$ partial derivatives provide a complete description of how $f$ varies locally — they tell the full story.

When all $n$ of them exist at a given point, $f$ earns a name for the property.

The Gradient

When $f$ is partially differentiable at $\mathbf{a}$, its $n$ partial derivatives can be collected into a single vector — the gradient.

The symbol $\nabla$ is called the nabla (or del) operator. Differential operators like $\nabla$ will be explored in more depth later in the course.

The gradient is the payoff for working in a linear vector space. When $f_{x_1}, \ldots, f_{x_n}$ are continuous, any directional derivative reduces to an inner product with the gradient:

This is the key trick: instead of computing a separate tortuous limit for every possible diagonal direction $\mathbf{v}$, we compute the gradient once — just $n$ partial derivatives — and recover any directional derivative for free via an inner product (dot product). The gradient packages all local directional information into a single vector.

Steepest Ascent and Descent

The gradient does more than package the partial derivatives — it points in a very specific geometric direction. Suppose $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$ (with $D$ open) is a scalar field with continuous partial derivatives $f_{x_1}, \ldots, f_{x_n}$, and let $\mathbf{a} \in D$ be a point where $\nabla f(\mathbf{a}) \neq \mathbf{0}$. Then $f$ has its steepest ascent at $\mathbf{a}$ in the direction $\nabla f(\mathbf{a})$, and its steepest descent in the opposite direction $-\nabla f(\mathbf{a})$.

Intuitively: standing on a hillside, the gradient points straight uphill along the steepest route, and its negative points straight downhill (the way a ball would roll). Every other direction trades some uphill progress for sideways motion.

This follows directly from the inner-product formula $\frac{\partial f}{\partial \mathbf{v}}(\mathbf{a}) = \langle \nabla f(\mathbf{a}),\, \mathbf{v} \rangle$: among all unit vectors $\mathbf{v}$, this inner product is largest when $\mathbf{v}$ is perfectly aligned with $\nabla f(\mathbf{a})$ and smallest when it points the opposite way. The one strict condition is $\nabla f(\mathbf{a}) \neq \mathbf{0}$ — at a point where the gradient vanishes, there is no preferred direction (the ground is flat).

Isolines

Alongside the gradient, another geometric object captures the shape of a scalar field: the set of points where $f$ takes a constant value.

Think of a topographic map: each contour traces the points at the same elevation. On a weather map, each isotherm connects places with the same temperature. More generally, isolines slice the domain into level sets on which $f$ is constant. (Note: A notation like $N_{f(\mathbf{a})=c}$ is just a formal way of saying “the specific isoline that passes through our base point $\mathbf{a}$”).

The gradient at a point $\mathbf{a}$ is always perpendicular to the isoline $N_{f(\mathbf{a}) = c}$ passing through $\mathbf{a}$.

To see why, pick any unit vector $\mathbf{v}$ that points along the isoline (a tangent vector). Elevation doesn’t change in that direction, so the directional derivative vanishes:

Since a zero inner product means two vectors are orthogonal, the steepest ascent $\nabla f(\mathbf{a})$ is forced to be perpendicular to the isoline.

Second-Order Partial Derivatives

The partial derivatives $f_{x_1}, \ldots, f_{x_n}$ of a scalar field $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$ are themselves scalar fields on $D$ — each one assigns to every point the rate of change of $f$ along the corresponding coordinate axis. We assume $D$ is an open set here, which simply means every point of $D$ has a bit of breathing room around it that still lies inside $D$ — no point sits right on the edge. That matters because computing $f_{x_i}(\mathbf{a})$ means peeking at $f$ just to either side of $\mathbf{a}$ along the $i$-th axis; if $\mathbf{a}$ were on the boundary, some of those nearby probe points would fall outside $D$ where $f$ isn’t defined, and the limit couldn’t be formed at all. Openness rules that case out, so the partial derivatives exist throughout $D$.

When all $n$ of these derived scalar fields are continuous, we give the situation its own name.

If these first-order partial derivatives are themselves partially differentiable, we can differentiate a second time — taking a partial derivative of a partial derivative.

All four expressions on the right denote exactly the same quantity — they are fully interchangeable, just different notations for the same second-order partial derivative (read them right-to-left like peeling an onion: first derive by $x_i$, then derive the result by $x_j$).

The intuition mirrors the 1D case. In 1D, the second derivative $f''(a)$ measures curvature — whether the slope is growing or shrinking as we move along the axis. On a hill, $f''$ tells you whether the climb is getting steeper or starting to level out. The same picture carries over to higher dimensions: an unmixed derivative $f_{x_i x_i}$ describes how the slope along the $i$-th axis is itself changing as we step further along $x_i$, and a mixed derivative $f_{x_j x_i}$ describes how a slope along one axis varies as we step along another.

Smoothness

The construction extends arbitrarily: differentiate a partial derivative once more to get a third-order partial derivative, again for fourth-order, and so on without limit. When all partial derivatives up to order $k$ exist and are continuous, we get a property worth naming.

These nested smoothness levels get their own family of names — the $C^k$ classes — each one packaging ”$f$ has $k$ continuous orders of derivatives” into a single label.

So $C^0(D)$ is just the continuous functions (no jumps) on $D$; $C^k(D)$ for $k \geq 1$ asks for $k$ continuous orders of partial derivatives; and $C^\infty(D)$ is the set of functions that remain differentiable (perfectly smooth) no matter how many times you differentiate them. Two milestones from this hierarchy matter most going forward: $f \in C^1(D)$ means all first-order partial derivatives are continuous, so we can assemble them into the gradient $\nabla f$ at every point of $D$; and $f \in C^2(D)$ means all second-order partial derivatives are continuous, which is exactly what is needed to assemble them into the Hessian.

In practice, we rarely care about the exact value of $k$ — we just want enough continuous derivatives for whatever theorem or computation we’re doing. That looser notion gets its own name.

The required $k$ depends on context — if a result needs second-order partial derivatives to be continuous, “smooth enough” means $C^2(D)$; if third-order derivatives are needed, $C^3(D)$, and so on. In most practical settings one simply assumes $f \in C^\infty(D)$ to avoid having to track the exact order.

Once $f$ reaches $C^2$ smoothness, a useful property kicks in — the order in which we take partial derivatives stops mattering.

Hessian Matrix

With the first-order partial derivatives, we organized the $n$ values $f_{x_1}, \ldots, f_{x_n}$ into a single vector — the gradient. Second-order partials are richer: for each pair $(x_i, x_j)$ there is one derivative $f_{x_j x_i}$, giving $n^2$ numbers in total. The natural way to organize them is as an $n \times n$ matrix. The pattern continues into higher orders — third-order partial derivatives need a three-index object with $n^3$ entries (a tensor), fourth-order ones live in $n^4$ entries, and so on — but at second order, this $n \times n$ matrix has its own name.

The entry in row $i$, column $j$ is $f_{x_i x_j}(\mathbf{x})$ — the rate at which the $j$-th partial derivative of $f$ changes as we step along $x_i$. The diagonal entries $f_{x_i x_i}$ are the pure second derivatives along each axis; the off-diagonal entries $f_{x_i x_j}$ (for $i \neq j$) are the mixed partials.

Since $f \in C^2(D)$, the mixed partials satisfy $f_{x_i x_j} = f_{x_j x_i}$ for all $(i, j)$ — so the Hessian is always symmetric.

Stationary Points and Extrema

The gradient and Hessian are not brand-new objects — they are the $n$-dimensional counterparts of the first and second derivatives we already know from 1D calculus, and they carry over the same roles.

In 1D, finding the extrema of a smooth function $f : \mathbb{R} \to \mathbb{R}$ follows a two-step recipe: first solve $f'(a) = 0$ to locate candidate points (bottoms of valleys, tops of peaks), then evaluate $f''(a)$ at those candidates — a positive value means the curve opens upward (local minimum), a negative value means it opens downward (local maximum), and zero leaves the test inconclusive.

The exact same logic plays out in $n$ dimensions, with the gradient playing the role of $f'$ and the Hessian playing the role of $f''$. The “zero-slope” condition now becomes $\nabla f(\mathbf{a}) = \mathbf{0}$ — every partial derivative vanishes at once. Points where this holds are called stationary points — defined formally in the optima chapter.

Multivariate Functions

Every definition so far — partial derivatives, gradient, Hessian, continuous partial differentiability, class $C^k$ — has been stated for scalar fields $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$. Extending them to multivariate functions is mechanical: a multivariate function $f : D \subseteq \mathbb{R}^n \to \mathbb{R}^m$ is nothing more than an $m$-tuple of scalar fields stacked vertically:

Each component function $f_i : D \subseteq \mathbb{R}^n \to \mathbb{R}$ is itself a scalar field — exactly the kind of object every earlier definition applied to. Every earlier property therefore lifts componentwise:

• Partially differentiable: $f$ is partially differentiable at $\mathbf{a} \in D$ (or on $D$) if and only if every component $f_i$ is partially differentiable at $\mathbf{a}$ (or on $D$).
• $k$-times continuously partially differentiable: $f$ is $k$-times continuously partially differentiable at $\mathbf{a}$ (or on $D$) if and only if every $f_i$ is.
• Class $C^k(D)$: $f$ belongs to $C^k(D)$ if and only if every $f_i$ belongs to $C^k(D)$.

Nothing new to prove — each property is simply applied to each component in turn.

One term gets its own label in the multivariate setting: when all partial derivatives of $f$ not only exist but are also continuous, we say $f$ is differentiable.

Jacobian Matrix

Once a multivariate function is differentiable — every component $f_i$ contributing $n$ continuous partial derivatives — the natural way to organize all of its first-order information is into a single matrix. For a scalar field we collected $n$ partials into the gradient vector. With $m$ components, each carrying $n$ partials, we now have $m \cdot n$ numbers in total, and they fit cleanly into an $m \times n$ matrix.

Reading the matrix row by row makes the structure obvious: the $i$-th row is exactly the gradient of the $i$-th component function $f_i$, written as a row vector. The Jacobian is therefore a vertical stack of component gradients — each row carrying the full first-order story of one output coordinate, and each column carrying the response of all outputs to a nudge in one input variable.

An alternative notation often seen in the literature is $Df(\mathbf{x})$, used interchangeably with $J_f(\mathbf{x})$.

As a useful special case, when $m = 1$ the function reduces to a scalar field and the Jacobian has only a single row — exactly the transposed gradient:

So the gradient and the Jacobian are not separate ideas: the gradient is just the Jacobian’s only row when there is only one output to track.

The deeper purpose of the Jacobian is to provide the best linear approximation of $f$ near $\mathbf{x}$. If $f$ is differentiable at $\mathbf{x}$, then for a small step $\mathbf{h}$:

with the approximation getting sharper as $\|\mathbf{h}\| \to 0$. This is the multivariate version of the slope picture used earlier to introduce the directional derivative: in 1D, $f(a + h) \approx f(a) + f'(a)\,h$ — the very same $\frac{\Delta y}{\Delta x} \approx f'(a)$ tangent line, just rearranged to predict the function value a small step past $a$. The scalar slope $f'(a)$ is replaced by the matrix $J_f(\mathbf{x})$, which acts on the displacement vector $\mathbf{h}$ to produce the predicted change in every output coordinate at once. A more detailed treatment follows in the discussion of coordinate transformations.

A particularly tidy special case: when $f$ is itself a linear function, $f(\mathbf{x}) = A\mathbf{x} + \mathbf{b}$ for some constant $m \times n$ matrix $A$ and constant vector $\mathbf{b}$, the Jacobian is just $A$ at every point of the domain. Each component $f_i$ is the linear combination $a_{i1}x_1 + \cdots + a_{in}x_n + b_i$, whose partial derivative with respect to $x_j$ is the constant $a_{ij}$, so the Jacobian’s $(i, j)$ entry is $a_{ij}$ everywhere. The linear approximation $f(\mathbf{x} + \mathbf{h}) \approx f(\mathbf{x}) + J_f(\mathbf{x})\,\mathbf{h}$ then stops being an approximation and becomes the exact equality $f(\mathbf{x} + \mathbf{h}) = f(\mathbf{x}) + A\,\mathbf{h}$ — a linear function coincides with its tangent linear map everywhere.

Calculation Rules

The Jacobian inherits a familiar set of computation rules — exactly the multivariate counterparts of the standard differentiation rules from 1D calculus. Sums differentiate term by term, scalars factor out, products obey a product rule, and compositions follow a chain rule. The only twist in higher dimensions is that the chain rule turns into a matrix multiplication of two Jacobians, so the order of multiplication now matters.

Let $f, g : D \subseteq \mathbb{R}^n \to \mathbb{R}^m$ be partially differentiable. Then for every $\mathbf{x} \in D$:

• Additivity: $J_{f+g}(\mathbf{x}) = J_f(\mathbf{x}) + J_g(\mathbf{x})$
• Homogeneity: $J_{\lambda f}(\mathbf{x}) = \lambda\, J_f(\mathbf{x})$ for all $\lambda \in \mathbb{R}$
• Product rule: $J_{f^\top g}(\mathbf{x}) = f(\mathbf{x})^\top J_g(\mathbf{x}) + g(\mathbf{x})^\top J_f(\mathbf{x})$

Additivity and homogeneity together say that the Jacobian behaves as a linear operator on partially differentiable functions — directly mirroring the linearity of the 1D derivative, $(f+g)' = f'+g'$ and $(\lambda f)' = \lambda f'$. The product rule echoes the Leibniz pattern $(fg)' = f'g + fg'$, but since $f$ and $g$ are vector-valued the natural scalar product is the inner product $f^\top g : D \to \mathbb{R}$, and each factor carries a transpose to match the row-vector shapes the Jacobian expects.

The remaining rule needs two functions whose dimensions are compatible for composition. Let $f : D \subseteq \mathbb{R}^n \to \mathbb{R}^m$ and $g : D' \subseteq \mathbb{R}^l \to \mathbb{R}^n$ with $g(D') \subseteq D$, so the composition $h = f \circ g : D' \to \mathbb{R}^m$ is well-defined. Then for every $\mathbf{x} \in D'$:

This is the composition rule, also widely known as the chain rule. The right-hand side is a matrix product: $J_f(g(\mathbf{x}))$ is $m \times n$, $J_g(\mathbf{x})$ is $n \times l$, and the result is $m \times l$ — exactly the shape required for $J_{f \circ g}(\mathbf{x})$. It is the direct generalization of the 1D chain rule $(f \circ g)'(x) = f'(g(x))\, g'(x)$, with scalar multiplication replaced by matrix multiplication.

Reading off the entry in row $i$ and column $j$ of $J_{f \circ g}(\mathbf{x})$ recovers the familiar partial-derivative form:

Here $i$ is the row index — the output component $h_i$ of the composition — and $j$ is the column index — the input variable $x_j$. The sum over $k$ runs through the $n$ intermediate outputs of $g$, which is exactly the inner product of the $i$-th row of $J_f(g(\mathbf{x}))$ with the $j$-th column of $J_g(\mathbf{x})$ — the row-times-column rule of matrix multiplication, written out one entry at a time.

Differential Operators

A differential operator is an abstract mapping that takes a function as input and returns another function as output, with the (partial) derivatives of the input doing the structural work. The operators in this section all eat a function defined on a domain $D \subseteq \mathbb{R}^n$ — a scalar field or a vector field, depending on the operator — and produce a new function as output. We assume throughout that every partial derivative of the input function exists and is continuous on $D$, so the constructions below are well-defined at every point.

A useful organizing idea before diving in: most of the operators that follow can be expressed as a single, more primitive operator — the nabla operator — combined with the standard vector products (scalar multiplication, inner product, cross product). Setting up that primitive carefully first makes everything else mechanical.

Nabla Operator

The nabla operator is the most fundamental differential operator on $\mathbb{R}^n$ — and one we have already been using implicitly: it is exactly the object that turned a scalar field into its gradient back in the gradient definition, where writing $\nabla f$ meant “stack the partial derivatives of $f$ into a column vector”. Pulling it out as a stand-alone operator simply makes that move explicit and lets the same primitive serve the divergence and the rotation below. It is best read as a formal column “vector” whose entries are the $n$ partial-derivative operators $\partial_1, \ldots, \partial_n$ — a column of operators, not numbers, waiting to be applied to a function.

Input: a function on $D$ — either a scalar field $f$ or a vector field $\boldsymbol{v}$, depending on which “vector multiplication” is used. Output: a new function whose shape depends on the multiplication.

By itself $\nabla$ is not a function and has no value at a point — it is purely a column of operators. It only produces a result once paired with an actual function via one of the standard vector products, and the product determines what the result looks like:

• Applying to a scalar field: $\nabla : (f: D \subseteq \mathbb{R}^n \to \mathbb{R}) \to (D \to \mathbb{R}^n),\ f \mapsto (\partial_1 f, \ldots, \partial_n f)^\top$ — think of $\nabla = (\partial_1, \ldots, \partial_n)^\top$ as a formal column vector of operators; applying it to $f$ means each slot acts via partial differentiation, giving exactly the gradient gradient.

• Inner product with a vector field: $\nabla \cdot : (\boldsymbol{v}: D \subseteq \mathbb{R}^n \to \mathbb{R}^n) \to (D \to \mathbb{R}),\ \boldsymbol{v} \mapsto \sum_i \partial_i v_i$ — pair each $\partial_i$ with the matching component $v_i$, apply, then sum, exactly like an inner product but application replaces multiplication; the result is the divergence divergence.

• Cross product with a vector field: $\nabla \times : (\boldsymbol{v}: D \subseteq \mathbb{R}^3 \to \mathbb{R}^3) \to (D \to \mathbb{R}^3),\ \boldsymbol{v} \mapsto (\partial_2 v_3 - \partial_3 v_2,\ \partial_3 v_1 - \partial_1 v_3,\ \partial_1 v_2 - \partial_2 v_1)^\top$ — exactly like a cross product but application replaces multiplication; the result is the rotation rotation.

Laplace Operator

Divergence

Rotation

Defined only in $\mathbb{R}^3$, since the cross product itself is exclusive to three dimensions.