Self-Information

Self-Information

Self-information (also called surprisal) quantifies how much information is gained when you learn that an event with probability $p$ has occurred.

$I(p) = -\log_2(p) \text{ bits}$

Intuition

• A certain event ($p = 1$) carries 0 bits of information — you already knew it would happen.
• A coin flip ($p = 0.5$) carries 1 bit — you need one yes/no question to resolve it.
• A rare event ($p = 0.125$) carries 3 bits — it takes three yes/no questions to pin down.

Log Base Choice

Using $\log_2$ gives information in bits. Using $\ln$ gives nats. This course uses bits throughout.

Example

$I(0.25) = -\log_2(0.25) = -\log_2(2^{-2}) = 2 \text{ bits}$

import math

def self_information(p):
    return -math.log2(p)

print(self_information(0.5))   # 1.0 bit
print(self_information(0.25))  # 2.0 bits

Your Task

Implement three equivalent functions:

• self_information(p) — returns $-\log_2(p)$
• information_content(p) — same formula
• surprise(p) — same formula (different name used in some literature)

All three should return results in bits.