Introduction

Why Information Theory?

Information theory, founded by Claude Shannon in 1948, is the mathematical foundation of modern communication, compression, and machine learning. Every time you send a message, compress a file, or train a neural network, you are applying information theory.

• Entropy — quantifies uncertainty and the fundamental limits of compression.
• Mutual information — measures statistical dependence, used in feature selection and causal inference.
• KL divergence and cross-entropy — the loss functions of modern deep learning.
• Channel capacity — the theoretical maximum throughput of any noisy communication system.
• Coding theory — bridges abstract math with practical data compression.

This course implements every concept from scratch in Python, giving you both the mathematical intuition and the computational tools.