Differential Relative Entropy

Why

We want relative entropy for densities.

What

The differential relative entropy of a second density with respect to the first density is the cross entropy of the second with respect to the first less the differential entropy of the first.

Notation

Let $f$ and $g$ be densities. The relative entropy of $f$ with respect to $g$ is

\[ h(g, f) - h(f) \]