Logistic Regression

Logistic Regression

Logistic regression is the standard model for binary classification. Instead of predicting a continuous value, it predicts the probability that an example belongs to class 1.

The Sigmoid Function

The sigmoid (logistic) function squashes any real number into the interval $(0, 1)$:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

Notation note: In ML, $\sigma$ conventionally denotes the sigmoid activation function. In statistics and finance, the same symbol $\sigma$ represents standard deviation or volatility — an entirely different quantity.

Key properties:

• $\sigma(0) = 0.5$
• $\sigma(z) \to 1$ as $z \to +\infty$
• $\sigma(z) \to 0$ as $z \to -\infty$

Numerical Stability

The naive formula 1 / (1 + exp(-z)) overflows when z is a large negative number (e.g., -1000), because exp(1000) exceeds floating-point range.

The fix: when $z \ge 0$, use the formula directly. When $z < 0$, rewrite as:

$\sigma(z) = \frac{e^z}{1 + e^z}$

This way the exponent is always non-positive, so exp() never overflows:

def sigmoid(z):
    if z >= 0:
        return 1 / (1 + math.exp(-z))
    else:
        ez = math.exp(z)
        return ez / (1 + ez)

Logistic Prediction

$\hat{p} = \sigma(wx + b) = \frac{1}{1 + e^{-(wx+b)}}$

Binary Cross-Entropy Loss

For binary classification with labels $y_i \in \{0, 1\}$, we minimise:

$\mathcal{L} = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \log(\hat{p}_i) + (1 - y_i) \log(1 - \hat{p}_i) \right]$

Use $\varepsilon = 10^{-15}$ inside the logarithms to avoid $\log(0)$.

Your Task

Implement:

• sigmoid(z) — the logistic function
• logistic_predict(x, w, b) — sigmoid of the linear combination
• binary_cross_entropy(y_pred, y_true) — average cross-entropy loss