Partial Differentiation

Consider a scalar field of the form:

$f : D \subseteq \mathbb{R}^n \to \mathbb{R}, \quad \mathbf{x} = (x_1, \ldots, x_n)^\top \mapsto f(\mathbf{x}) = f(x_1, \ldots, x_n)$

This is a function that takes a point in $n$ -dimensional space and returns a single real number — like a temperature field in a room, or an elevation map over terrain.

Directional Derivative

Recall from one-dimensional calculus that the derivative of $f$ at a point $a$ is:

$f'(a) = \lim_{h \to 0} \frac{f(a + h) - f(a)}{h}$

This is the slope of the function at $a$ : how much does $f$ change when we take a tiny step of size $h$ away from $a$ ? Geometrically, it is the same intuition behind $\tan(\alpha) = \frac{\sin(\alpha)}{\cos(\alpha)}$ — the slope of a line is simply the ratio of the vertical rise to the horizontal run, i.e., $\frac{\Delta y}{\Delta x} = \frac{f(a+h) - f(a)}{h}$ .

In multiple dimensions, the same question becomes richer: we can step away from a point in infinitely many directions. This is where the directional derivative comes in — it tells us the rate of change of $f$ when we stand at a point $\mathbf{a}$ and look in a specific direction $\mathbf{v}$ .

Here, $\mathbf{a} \in D$ is the base point — the location in the domain where we want to measure how fast $f$ is changing. Think of it as your current position on a terrain map: you first choose where you stand, then you choose which way to look. Crucially, the directional derivative is a local quantity — it captures the behavior of $f$ specifically at $\mathbf{a}$ , not globally.

For each unit vector $\mathbf{v}$ (with $\|\mathbf{v}\| = 1$ ), we take a tiny step of size $h$ from $\mathbf{a}$ in the direction $\mathbf{v}$ , measure the change in $f$ , and divide by the step size — the same slope formula as before, now generalized to any direction in $n$ -dimensional space:

$\frac{f(\mathbf{a} + h\mathbf{v}) - f(\mathbf{a})}{h}$

As $h \to 0$ , the step size becomes infinitesimally small, and we recover the instantaneous rate of change in the chosen direction.

The directional derivative of a scalar field $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$ at a point $\mathbf{a} \in D$ in the direction of a unit vector $\mathbf{v}$ (with $\|\mathbf{v}\| = 1$ ) is:

$\frac{\partial f}{\partial \mathbf{v}}(\mathbf{a}) = \partial_\mathbf{v} f(\mathbf{a}) = f_\mathbf{v}(\mathbf{a}) = \lim_{h \to 0} \frac{f(\mathbf{a} + h\mathbf{v}) - f(\mathbf{a})}{h}$

All three notations — $\frac{\partial f}{\partial \mathbf{v}}(\mathbf{a})$ , $\partial_\mathbf{v} f(\mathbf{a})$ , and $f_\mathbf{v}(\mathbf{a})$ — refer to the same quantity: the rate of change of $f$ at the base point $\mathbf{a}$ along the direction $\mathbf{v}$ .

Requiring $\mathbf{v}$ to be a unit vector ensures that the step $h\mathbf{v}$ has length exactly $|h|$ , making the derivative a pure measure of directional rate of change, independent of the magnitude of $\mathbf{v}$ .

Since $\mathbf{v}$ can point in any direction on the unit sphere in $\mathbb{R}^n$ , there are infinitely many directional derivatives at any given point $\mathbf{a}$ . In principle, this limit may not exist for every combination of $f$ , $\mathbf{a}$ , and $\mathbf{v}$ . However, throughout this course we will work with sufficiently smooth functions and always assume the limit exists — the “happy scenario.”

Partial Derivatives

Having infinitely many directions to check is impractical. The saving insight is that we are working in a linear vector space: any direction $\mathbf{v}$ can be expressed as a linear combination of the $n$ vectors of the standard basis (also called the canonical basis) of $\mathbb{R}^n$ . For smooth functions, this means the behavior of $f$ in every direction is fully determined by its behavior along the $n$ coordinate directions — one per argument of the function. We do not need to sweep through all directions; the $n$ coordinate directions are enough.

Substituting $\mathbf{v} = \mathbf{e}_i$ into the directional derivative formula means we step from $\mathbf{a}$ by a tiny amount $h$ purely along the $i$ -th axis, holding all other coordinates fixed. The resulting limit is the rate of change of $f$ with respect to its $i$ -th argument $x_i$ .

The partial derivative of $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$ at a point $\mathbf{a} \in D$ with respect to the variable $x_i$ is:

$\frac{\partial f}{\partial x_i}(\mathbf{a}) = \partial_i f(\mathbf{a}) = f_{x_i}(\mathbf{a}) = \lim_{h \to 0} \frac{f(\mathbf{a} + h\mathbf{e}_i) - f(\mathbf{a})}{h}, \quad i \in \{1, \ldots, n\}$

As always, we assume this limit exists. Each partial derivative answers a single focused question: how fast does $f$ change when only the $i$ -th coordinate of $\mathbf{a}$ is nudged, while all others stay fixed? Together, the $n$ partial derivatives provide a complete description of how $f$ varies locally — they tell the full story.

The Gradient

Once all $n$ partial derivatives at $\mathbf{a}$ exist, we say $f$ is partially differentiable at $\mathbf{a}$ . We can then collect these derivatives into a single vector — the gradient.

The gradient of $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$ at a point $\mathbf{a} \in D$ is the column vector of all partial derivatives:

$\nabla f(\mathbf{a}) = \begin{pmatrix} f_{x_1}(\mathbf{a}) \\ \vdots \\ f_{x_n}(\mathbf{a}) \end{pmatrix} = \operatorname{grad} f(\mathbf{a})$

The gradient $\nabla f(\mathbf{a}) \in \mathbb{R}^n$ lives in the same space as the input — not the output. So for a scalar field $f : \mathbb{R}^2 \to \mathbb{R}$ , the gradient at any point is a 2D vector; for $f : \mathbb{R}^3 \to \mathbb{R}$ , it’s a 3D vector.

This means we can evaluate $\nabla f$ at every point in the domain, producing a gradient field $\mathbf{a} \mapsto \nabla f(\mathbf{a})$ — a vector field on $D$ . Think of it as an arrow attached to each input point, pointing in the direction of steepest ascent at that location.

The symbol $\nabla$ is called the nabla (or del) operator. Differential operators like $\nabla$ will be explored in more depth later in the course.

The gradient is the payoff for working in a linear vector space. When $f_{x_1}, \ldots, f_{x_n}$ are continuous, any directional derivative reduces to a dot product with the gradient:

$\frac{\partial f}{\partial \mathbf{v}}(\mathbf{a}) = \langle \nabla f(\mathbf{a}),\, \mathbf{v} \rangle$

This is the key trick: instead of computing a separate limit for every possible direction $\mathbf{v}$ , we compute the gradient once — just $n$ partial derivatives — and recover any directional derivative for free via an inner product. No more tortuous limit computations for each new direction; the gradient packages all local directional information into a single vector.

Let $f(x, y, z) = e^{-z}\sin x + y^2$ . Using standard differentiation rules, the three partial derivatives are:

$\frac{\partial f}{\partial x} = e^{-z}\cos x, \qquad \frac{\partial f}{\partial y} = 2y, \qquad \frac{\partial f}{\partial z} = -e^{-z}\sin x$

Assembling them into the gradient:

$\nabla f(x, y, z) = \begin{pmatrix} e^{-z}\cos x \\ 2y \\ -e^{-z}\sin x \end{pmatrix}$

Steepest Ascent and Descent

The gradient does more than package the partial derivatives — it points in a very specific geometric direction. Suppose $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$ is a scalar field with continuous partial derivatives $f_{x_1}, \ldots, f_{x_n}$ , and let $\mathbf{a} \in D$ be a point where $\nabla f(\mathbf{a}) \neq \mathbf{0}$ . Then $f$ has its steepest ascent at $\mathbf{a}$ in the direction $\nabla f(\mathbf{a})$ , and its steepest descent in the opposite direction $-\nabla f(\mathbf{a})$ .

Intuitively: standing on a hillside, the gradient points straight uphill along the steepest route, and its negative points straight downhill. Every other direction trades some uphill progress for sideways motion.

This follows directly from the dot-product formula $\frac{\partial f}{\partial \mathbf{v}}(\mathbf{a}) = \langle \nabla f(\mathbf{a}),\, \mathbf{v} \rangle$ : among all unit vectors $\mathbf{v}$ , this inner product is largest when $\mathbf{v}$ is aligned with $\nabla f(\mathbf{a})$ and smallest when it points the opposite way. The one strict condition is $\nabla f(\mathbf{a}) \neq \mathbf{0}$ — at a point where the gradient vanishes, there is no preferred direction.

Isolines

Alongside the gradient, another geometric object captures the shape of a scalar field: the set of points where $f$ takes a constant value.

Let $f : D \subseteq \mathbb{R}^n \to \mathbb{R}$ be a scalar field. For a value $c \in f(D) \subseteq \mathbb{R}$ , the isoline (also called contour line or level set) of $f$ at level $c$ is:

$N_c = \{\, \mathbf{x} \in D \mid f(\mathbf{x}) = c \,\}$

Think of a topographic map: each contour traces the points at the same elevation. On a weather map, each isotherm connects places with the same temperature. More generally, isolines slice the domain into level sets on which $f$ is constant.

The gradient at a point $\mathbf{a}$ is always perpendicular to the isoline $N_{f(\mathbf{a}) = c}$ passing through $\mathbf{a}$ . The reasoning is short. By definition, $f$ takes the constant value $c = f(\mathbf{a})$ everywhere on the isoline. Now pick any unit vector $\mathbf{v}$ that points along the isoline at $\mathbf{a}$ — that is, a direction you could step in to stay on the isoline, the way a hiker walks along a contour line without gaining or losing elevation. Such a $\mathbf{v}$ is called a tangent vector to the isoline at $\mathbf{a}$ . Since $f$ does not change as we move in this direction, the directional derivative along $\mathbf{v}$ must be zero. Using the dot-product formula for the directional derivative:

$\langle \nabla f(\mathbf{a}),\, \mathbf{v} \rangle = \frac{\partial f}{\partial \mathbf{v}}(\mathbf{a}) = 0$

and a zero inner product means the two vectors are orthogonal. Since this holds for every tangent direction of the isoline, $\nabla f(\mathbf{a})$ is perpendicular to the isoline as a whole. Back to the hillside: walking along a contour keeps your elevation fixed, and the uphill direction is exactly perpendicular to the contour you are standing on.

$f: [-2,2] \times [-2,2] \to \mathbb{R}, \quad f(x,y) = x^2 + y^2, \quad \nabla f(x,y) = \begin{pmatrix} 2x \\ 2y \end{pmatrix}$

In the example above, the 2D floor is the domain of $f$ : the heatmap encodes $f(x,y)$ as color, the dark rings are isolines (level sets $f(x,y) = c$ ), and the red arrows are the gradient field $\nabla f$ . This is where $f$ , its isolines, and its gradient actually live. The 3D bowl is the graph $z = f(x,y)$ — a visualization aid, not a separate object. The same isolines and gradient arrows are lifted onto it: the rings become horizontal cross-sections of the bowl, and the arrows become tangent vectors pointing in the steepest-ascent direction along the surface.

Arrow length. The arrows are short near the center and long at the boundary — not by coincidence. For $f(x,y) = x^2 + y^2$ :

$\|\nabla f\| = \sqrt{(2x)^2 + (2y)^2} = \sqrt{4x^2 + 4y^2} = 2\sqrt{x^2 + y^2} = 2r$

The gradient magnitude is just twice the radial distance. Geometrically: near the origin the bowl is nearly flat, so there’s barely any slope to point along; near the rim it’s steep, so the gradient is large. Exactly at the origin $\|\nabla f\| = 0$ — you’re at the minimum, there’s no downhill direction, the gradient has nothing to say.

Arrow direction. Every arrow points straight outward, perpendicular to the isoline it sits on. That’s not specific to this example — it’s always true: the gradient is orthogonal to the level set. If you stood on the 3D bowl and asked “which way is straight up the slope?”, the answer is exactly where the lifted arrow points.