Multivariate Calculus

Differentials

• $df=f_{x}dx+f_{y}dy+f_{z}dz$
• $df\ne \mathrm{\Delta }f$
• $\mathrm{\Delta }f=f_x\mathrm{\Delta }x+f_y\mathrm{\Delta }y+f_z\mathrm{\Delta }z$
• $df$ is used to encode infinitesimal changes
• used to act as a placegolder value
• divide wrt time to get rate of change $\to$ CHAIN RULE

Chain Rule with More Variables

Let $w=f(x,y)$ when $x(u,v),y(u,v)$ then,

$$dw=f_{x}dx+f_{y}dy=(f_x{x}_u+f_y{y}_u)du+(f_x{x}_v+f_y{y}_v)dv=\frac{\partial f}{\partial u}du+\frac{\partial f}{\partial v}dv$$

Gradient Vector

$$\frac{dw}{dt}=w_x\frac{dx}{dt}+w_y\frac{dy}{dt}+w_z\frac{dz}{dt}=\overrightarrow{\mathrm{\nabla }}w.\frac{d\overrightarrow{r}}{dt}$$

Note: $\overrightarrow{\mathrm{\nabla }}w\perp \text{ level surfaces}$ (tangent to the level surface at any given point)

Directional Derivatives

$$\frac{dw}{ds}{|}_{\hat{u}}=\overrightarrow{\mathrm{\nabla }}w\cdot \frac{d\overrightarrow{r}}{ds}=\overrightarrow{\mathrm{\nabla }}w\cdot \hat{u}$$

Implications

Direction of $\overrightarrow{\mathrm{\nabla }}w$ is the direction of fastest increase of $w$

Lagrange Multipliers

Goal: minima/maximize a multi-variable function ($min/maxf(x,y,z)$) where $x,y,z$ are not independent and $\mathrm{\exists }$ $g(x,y,z)=c$.

These can be obtained on combining the given restraints with the following.

$$\overrightarrow{\mathrm{\nabla }}f=\lambda \overrightarrow{\mathrm{\nabla }}g$$

Basic idea: to find $(x,y)$ where the level curves of $f$ and $g$ are tangent to each other ($\overrightarrow{\mathrm{\nabla }}f\parallel \overrightarrow{\mathrm{\nabla }}g$).

Note: Take care that the point is indeed a maxima or minima as required and not just a saddle point (second derivative test won't be applicable so be vigilant).

Functions	Example	Value	First derivative	Second derivative
$f:\mathbb{R}\to \mathbb{R}$	$x^2$	$\mathbb{R}$	$\mathbb{R}$	$\mathbb{R}$
$f:{\mathbb{R}}^d\to \mathbb{R}$	loss function	$\mathbb{R}$	${\mathbb{R}}^d[\text{gradient}]$	${\mathbb{R}}^{d\times d}[\text{hessian}]$
$f:{\mathbb{R}}^d\to {\mathbb{R}}^p$	neural net layer	${\mathbb{R}}^p$	${\mathbb{R}}^{d\times p}[\text{jacobian}]$	${\mathbb{R}}^{d\times p\times p}$

Gradient

$${\mathrm{\nabla }}_{x}f(x)={\mathrm{\nabla }}_{x}f(x_1,\cdots ,x_d)=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ ⋮\\ \frac{\partial f(x)}{\partial x_d}\end{array}\right]$$

$${\mathrm{\nabla }}_{A}f(A)=\left[\begin{array}{cc}\frac{\partial f(A)}{\partial A_{11}}& \cdots \\ ⋮& ⋮\\ \cdots & \frac{\partial f(A)}{\partial A_{mn}}\end{array}\right]$$

Hessian

We have $f:{\mathbb{R}}^d\to {\mathbb{R}}^p$ thus, $f(x_1,\cdots ,x_d)=\left[\begin{array}{c}{f}_1(x_1,\cdots ,x_d)\\ ⋮\\ f_p(x_1,\cdots ,x_d)\end{array}\right]$

Note: Hessians are square-symmetric matrices.

$${\mathrm{\nabla }}_x^{2}f(x)=\left[\begin{array}{ccc}\frac{{\partial }^{2}f(x)}{\partial x_1^2}& \frac{{\partial }^{2}f(x)}{\partial x_1{x}_2}& \cdots \\ ⋮& \ddots & ⋮\\ ⋮& \cdots & \frac{{\partial }^{2}f(x)}{\partial x_n^2}\end{array}\right]$$

Jacobian

$$J=\left[\begin{array}{ccc}{\displaystyle \frac{\partial \mathbf{f}}{\partial x_1}}& \cdots & {\displaystyle \frac{\partial \mathbf{f}}{\partial x_d}}\end{array}\right]=\left[\begin{array}{c}{\mathrm{\nabla }}^{\mathrm{T}}{f}_1\\ ⋮\\ {\mathrm{\nabla }}^{\mathrm{T}}{f}_p\end{array}\right]=\left[\begin{array}{ccc}{\displaystyle \frac{\partial f_1}{\partial x_1}}& \cdots & {\displaystyle \frac{\partial f_1}{\partial x_n}}\\ ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial f_p}{\partial x_1}}& \cdots & {\displaystyle \frac{\partial f_p}{\partial x_d}}\end{array}\right]$$

where ${\mathrm{\nabla }}^{\mathrm{T}}{f}_i$ is the transpose (row vector) of the gradient of the $i$ component.

Examples

• ${\mathrm{\nabla }}_x{b}^{T}x=b$
• ${\mathrm{\nabla }}_x^2{b}^{T}x=0$
• ${\mathrm{\nabla }}_x{x}^{T}Ax=2Ax$, if $A$ is symmetric
• ${\mathrm{\nabla }}_x^2{x}^{T}Ax=2A$, if $A$ is symmetric