Gradient Descent

Gradient Descent

Gradient descent is the optimization algorithm that trains linear regression. It iteratively adjusts $w$ and $b$ to minimise the MSE loss.

The Gradients

Given predictions $\hat{y}_i = wx_i + b$ and true labels $y_i$, the gradients of MSE with respect to $w$ and $b$ are:

$\frac{\partial \text{MSE}}{\partial w} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i) x_i$

$\frac{\partial \text{MSE}}{\partial b} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)$

Parameter Update

After computing the gradients, each parameter is updated by a small step in the negative gradient direction:

$w \leftarrow w - \alpha \cdot \nabla_w$

$b \leftarrow b - \alpha \cdot \nabla_b$

where $\alpha$ is the learning rate — a small positive number like $0.01$.

Training Loop

Repeat for many iterations:

1. Compute predictions $\hat{y}$
2. Compute gradients $\nabla_w$, $\nabla_b$
3. Update $w$ and $b$

Your Task

Implement:

• gradient_w(x, y_pred, y_true) — gradient of MSE w.r.t. $w$
• gradient_b(y_pred, y_true) — gradient of MSE w.r.t. $b$
• update_param(param, grad, lr) — one gradient descent step