Every student studying a field related to optimization should be able to work with differentials. They are simpler to apply than (partial) derivatives and strictly more expressive, especially when generalizing to matrix functions. We will motivate them in this post, following the book Matrix Differential Calculus.

In university of any engineering degree, they teach you the partial derivatives of a function. It tells you the rate of change between an input and an output of the function and uses the convention to represent them as $\mathit{\partial 𝑦}∕\mathit{\partial 𝑥}$. That works for functions with scalar input and scalar output, but does not generalize well for vector/matrix calculus. Also this convention is rather weird, as it introduces a fraction notation, while not being a real fraction (your are not allowed to move $\mathit{\partial 𝑥}$ out).

Instead, differentials use a different approach. They represent how a change of the input $\text{d}x$ should be transformed to arrive at the linear part of the output $\text{d}y$. Linear, because the term is only allowed to depend on $\text{d}x$ once, for example for $\varphi (x)=x^2$

the first differential is given as $\text{d}\varphi =2x\text{d}x$.

The function parameters may be scalar, vector or matrix, for demonstration purpose we will concentrate on scalar functions with matrix parameters, often encountered in machine learning context and denote them as $\varphi (F)$.

For the beginning, we start with a motivational example. Let’s consider the differential of the matrix inverse function $\text{d}{X}^{-1}$ for an invertible matrix $X$. We work with the identity $X^{-1}X=I$ and differentiate

Hence, bringing the part of equation involving $\text{d}X$ to the RHS and right-multiplying $X^{-1}$, we do get

So this tells us, that increasing by $\text{d}X$, increases the output by $-X^{-1}(\text{d}X)X^{-1}$ for the linear part. We cannot hope to represent this relation with derivatives so easily, as we would have to represent the partial derivates for all input/output combinations.

Another way to express the relation between first-order derivatives and first-order differentials is the following identity

with the Jacobian $J(x)$ describing the change at some coordinate $x$. To rewrite a differential to a derivative, we need to order elements such that we can express the Jacobian in the LHS relation above.

Comparing this to the approach of many machine learning frameworks, we see the similarity to vector-Jacobian product (vJp) to transform a gradient vector in a chain of Jacobian transformations. Sometimes, respresenting the transformation as a linear map is hard, and we can represent the vJp in a more general form for differentials.

We will directly go in media res and introduce differentials of matrix calculus. Many rules are straightforward and known from derivative rules. We have that for $A$ constant $\text{d}A=0$, and that the differential is a linear operator

The matrix product follows from product rule

Further, one can also prove that

and hence (following from log rule)

Folowing the identification theorem, we can define the chain rules for two functions $Z=f(Y)$ and $Y=g(X)$ as

resulting from $\text{d}Z=A(Y)\text{d}Y$ and $\text{d}Y=B(X)\text{d}X$.

Producing more rules, we can derive Hessian from the second differential. It is straightforward defines as taking the differential, of a differential: $\text{d}(\text{d}f(X))$. The third differential is again defines, as taking the differential of the second differential, and so on $\dots$.

Similar to the identification of the Jacobian matrix, we can also identify the Hessian from the second differential. We formally define the Hessian through the differential of the Jacobian matrix parametrized by $x$. Say $\text{d}\varphi =J^{⊺}(x)\text{d}x$ and $\text{d}J=H{(x)}^{⊺}\text{d}x$, then we have

We used in $\text{(⋆)}$ that ${\text{d}}^{2}x=0$.

Hence, if we arrive (after a length derivation) at some representation ${\text{d}}^2\varphi ={(\text{d}x)}^{⊺}B(x)\text{d}x$, then we know that the Hessian is $H=(B(x)+B{(x)}^{⊺})∕2$.

We finish with three examples of matrix functions taken from the book. The functions take all matrices as input:

1.

Let $\varphi (X)=\mathrm{tr}{X}^{⊺}\mathit{𝐴𝑋}$, then we have $$\begin{array}{llll}\hfill \text{d}\varphi & =\text{d}(\mathrm{tr}{X}^{⊺}\mathit{𝐴𝑋})\stackrel{\star }{=}\mathrm{tr}\text{d}(X^{⊺}\mathit{𝐴𝑋})& \hfill & \\ \hfill & \stackrel{\star \star }{=}\mathrm{tr}{(\text{d}X)}^{⊺}\mathit{𝐴𝑋}+\mathrm{tr}{X}^{⊺}A\text{d}X=\mathrm{tr}{X}^{⊺}(A+A^{⊺})\text{d}X& \hfill & \end{array}$$

We use the trace rule from previous section in

$\text{(⋆)}$, and the product rule in $\text{(⋆⋆)}$. The second differentials follow from

$$\begin{array}{lll}\hfill {\text{d}}^2\varphi =\text{d}\mathrm{tr}{X}^{⊺}(A+A^{⊺})\text{d}X=\mathrm{tr}\left\{{(\text{d}X)}^{⊺}(A+A^{⊺})\text{d}X\right\}& & \hfill \end{array}$$

2.

Let $\varphi (X)=\mathrm{tr}{X}^k$, then for $k\ge 1$ $$\begin{array}{llll}\hfill \text{d}\varphi & \stackrel{\star }{=}\mathrm{tr}(\text{d}X)X^{k-1}+\mathrm{tr}X(\text{d}X)X^{k-2}+\cdots +\mathrm{tr}{X}^{k-1}\text{d}X& \hfill & \\ \hfill & \stackrel{\star \star }{=}k\mathrm{tr}{X}^{k-1}\text{d}X& \hfill & \end{array}$$

We again apply the product rule for k-packets in $\text{(⋆)}$, and the known reordering in trace argument for $\text{(⋆⋆)}$. For $k\ge 2$

$$\begin{array}{lll}\hfill {\text{d}}^2\varphi =k\mathrm{tr}(\text{d}{X}^{k-1})\text{d}X=k\sum _{k=0}^{K-2}\mathrm{tr}{X}^k(\text{d}X)X^{K-2-k}\text{d}X.& & \hfill \end{array}$$

3.

Let $F(X)=A{X}^{-1}B$, then $$\begin{array}{lll}\hfill \text{d}F\stackrel{\star }{=}A(\text{d}{X}^{-1})B=-A{X}^{-1}(\text{d}X)X^{-1}B& & \hfill \end{array}$$

We applied the differential of an inverse in $\text{(⋆)}$, we have derived it in the first section.

$$\begin{array}{llll}\hfill {\text{d}}^{2}F& \stackrel{\star }{=}-A(\text{d}{X}^{-1})(\text{d}X)X^{-1}B-A{X}^{-1}(\text{d}X)(\text{d}{X}^{-1})B& \hfill & \\ \hfill & \stackrel{\star \star }{=}2A{X}^{-1}(\text{d}X)X^{-1}(\text{d}X)X^{-1}B.& \hfill & \end{array}$$

We cannot identify the Hessian matrix from second differentials, because the vec and Kronecker operator are missing.