Neural Network Forward Pass

Neural Network Forward Pass

A neural network layer takes an input vector $\mathbf{x}$ and applies a weight matrix $W$ and bias vector $\mathbf{b}$:

$\mathbf{z} = W\mathbf{x} + \mathbf{b}$

Each row $W_i$ contains the weights for one neuron:

$z_i = W_i \cdot \mathbf{x} + b_i = \sum_j W_{ij} x_j + b_i$

Then an activation function $\sigma$ (sigmoid or another non-linearity — not to be confused with the $\sigma$ used for standard deviation in statistics) is applied element-wise:

$\mathbf{a} = \sigma(\mathbf{z})$

Two-Layer Network

A network with two layers computes:

$\mathbf{a}^{(1)} = \text{ReLU}(W^{(1)}\mathbf{x} + \mathbf{b}^{(1)})$ $\hat{y} = W^{(2)}\mathbf{a}^{(1)} + b^{(2)}$

Your Task

Implement:

• layer_output(x, W, b) → list: $[W_i \cdot x + b_i]$ for each neuron row $W_i$
• relu_layer(x, W, b) → apply ReLU to each element of layer_output(x, W, b)