Entropy

Definitions

The Shannon information content of an outcome $x$ , measured in bits, is defined to be:

$h (x) = \log_{2} \frac{1}{P (x)}$

The entropy of an ensemble $X$ is defined to be the average Shannon information content of an outcome:

$H (X) \equiv \sum_{x \in A_{X}} P (x) \log \frac{1}{P (x)}$

Entropy is 0 when the outcome is deterministic, and maximized with value $\log (| A_{X} |)$ when the outcomes are uniformly distributed.

The joint entropy of two ensembles $X, Y$ is:

$H (X, Y) \equiv \sum_{x, y \in A_{x} A_{y}} P (x, y) \log \frac{1}{P (x, y)}$

Entropy is additive if the ensembles are independent:

$H (X, Y) = H (X) + H (Y)$

Entropy is decomposable.