Multilayer Perceptron · Module 2

Forward Propagation

How the values flow in the network

Calculate the first hidden layer all at once:

z^{(1)}=W^{(1)}x+b^{(1)}

\begin{bmatrix}z_1^{(1)}\\[8pt]z_2^{(1)}\\[8pt]z_3^{(1)}\\[8pt]z_4^{(1)}\end{bmatrix}

4 × 1

\begin{bmatrix}w_{11}^{(1)}&w_{12}^{(1)}&w_{13}^{(1)}&w_{14}^{(1)}&w_{15}^{(1)}\\[8pt]w_{21}^{(1)}&w_{22}^{(1)}&w_{23}^{(1)}&w_{24}^{(1)}&w_{25}^{(1)}\\[8pt]w_{31}^{(1)}&w_{32}^{(1)}&w_{33}^{(1)}&w_{34}^{(1)}&w_{35}^{(1)}\\[8pt]w_{41}^{(1)}&w_{42}^{(1)}&w_{43}^{(1)}&w_{44}^{(1)}&w_{45}^{(1)}\end{bmatrix}

4 × 5

\begin{bmatrix}x_1\\[8pt]x_2\\[8pt]x_3\\[8pt]x_4\\[8pt]x_5\end{bmatrix}

5 × 1

\begin{bmatrix}b_1^{(1)}\\[8pt]b_2^{(1)}\\[8pt]b_3^{(1)}\\[8pt]b_4^{(1)}\end{bmatrix}

4 × 1

a^{(1)}=f\!\left(z^{(1)}\right)

This 4 × 1 activation vector is the input to the next layer: $\quad z^{(2)}=W^{(2)}a^{(1)}+b^{(2)}$ .

z=W^{(4)}a^{(3)}+b^{(4)}\qquad\longrightarrow\qquad p=\sigma(z)=\frac{1}{1+e^{-z}}

In both modules, $f$ is the activation function applied to each hidden neuron. The output uses sigmoid.