Cellular Automata From First Principles 19: Entropy and Information

Mon, 10 Aug 2026 18:52:00 +0100

Cellular Automata From First Principles 19: Entropy and Information

A row with almost all zeros is highly predictable.

A row containing a balanced mixture of zeros and ones is less predictable.

Shannon entropy gives us a precise way to measure that uncertainty.

Binary entropy

If a binary state contains a fraction p of ones, then:

H = -p log2(p) - (1-p) log2(1-p)

In Python:

import math
import numpy as np


def binary_entropy(state):
    p = float(np.mean(state))

    if p in (0.0, 1.0):
        return 0.0

    return -(p * math.log2(p) + (1 - p) * math.log2(1 - p))

The maximum is 1 bit when zeros and ones occur equally often.

Entropy on Programmer.ie: Modern AI programming

Cellular Automata From First Principles 19: Entropy and Information

Cellular Automata From First Principles 19: Entropy and Information

Binary entropy