Discrete Entropy

Definition

The entropy of a distribution is the expectation of the negative logarithm of the distribution under the distribution. It is sometimes called the discrete entropy to distinguish it with another related topic.¹

Notation

Let $A$ be a finite set. Let $p:A \to \R $ be a distribution. The entropy of $p$ is

\[ -\sum_{a \in A} p(a) \log(p(a)). \]

Properties

Let $x: \Omega \to V$ be a discrete random variable.

1. $H(x) \geq 0$
2. $H(f(x)) \leq H(x)$
3. For invertible $g$, $H(g(x)) \leq H(x)$

1. Future editions may not forward reference differential entropy. ↩︎