Multilayer Perceptron · Module 3

Backpropagation

Highlighted forward pass: a fully connected 5–4–3–2–1 networkx₃ → a₁⁽¹⁾ → a₁⁽²⁾ → a₁⁽³⁾ → z. Highlighted lines trace one route; all other connections remain visible. The output logit z passes through sigmoid to produce the prediction.Input layer5 featuresHidden layer 14 neuronsHidden layer 23 neuronsHidden layer 32 neuronsOutput layer1 neuronw₁₁⁽¹⁾w₁₂⁽¹⁾w₁₃⁽¹⁾w₁₄⁽¹⁾w₁₅⁽¹⁾w₁₁⁽²⁾w₁₂⁽²⁾w₁₃⁽²⁾w₁₄⁽²⁾w₁₁⁽³⁾w₁₂⁽³⁾w₁₃⁽³⁾w₁₁⁽⁴⁾w₁₂⁽⁴⁾x₁x₂x₃x₄x₅a₁⁽¹⁾a₂⁽¹⁾a₃⁽¹⁾a₄⁽¹⁾a₁⁽²⁾a₂⁽²⁾a₃⁽²⁾a₁⁽³⁾a₂⁽³⁾zz = logitσ(z)σ(z) = p

Binary cross-entropy loss

L=−[y^ln⁡(p)+(1−y^)ln⁡(1−p)]L=-\left[\hat y\ln(p)+(1-\hat y)\ln(1-p)\right]

The loss LL compares our prediction pp with the ground truth y^∈{0,1}\hat y\in\{0,1\}. We want to find how changing each weight changes the loss.

Start with one output weight

z=w11(4)a1(3)+w12(4)a2(3)+b1(4),p=σ(z)z=w_{11}^{(4)}a_1^{(3)}+w_{12}^{(4)}a_2^{(3)}+b_1^{(4)},\qquad p=\sigma(z)

Follow w11(4)→z→p→Lw_{11}^{(4)}\to z\to p\to L. The chain rule gives:

∂L∂w11(4)=∂L∂p⋅∂p∂z⋅∂z∂w11(4)\frac{\partial L}{\partial w_{11}^{(4)}}=\frac{\partial L}{\partial p}\cdot\frac{\partial p}{\partial z}\cdot\frac{\partial z}{\partial w_{11}^{(4)}}

Calculate each derivative

∂L∂p=−y^p+1−y^1−p=p−y^p(1−p)\frac{\partial L}{\partial p}=-\frac{\hat y}{p}+\frac{1-\hat y}{1-p}=\frac{p-\hat y}{p(1-p)}
p=11+e−z⟹∂p∂z=e−z(1+e−z)2=p(1−p)p=\frac{1}{1+e^{-z}}\qquad\Longrightarrow\qquad\frac{\partial p}{\partial z}=\frac{e^{-z}}{(1+e^{-z})^2}=p(1-p)
∂z∂w11(4)=a1(3)\frac{\partial z}{\partial w_{11}^{(4)}}=a_1^{(3)}

The incoming activations, other output weight, and bias are held fixed when differentiating with respect to this weight.

Put them together

∂L∂w11(4)=(−y^p+1−y^1−p)⋅p(1−p)⋅a1(3)=[−y^(1−p)+(1−y^)p]a1(3)=(p−y^) a1(3)\begin{aligned}\frac{\partial L}{\partial w_{11}^{(4)}}&=\left(-\frac{\hat y}{p}+\frac{1-\hat y}{1-p}\right)\cdot p(1-p)\cdot a_1^{(3)}\\[12pt]&=\left[-\hat y(1-p)+(1-\hat y)p\right]a_1^{(3)}\\[12pt]&=\boxed{(p-\hat y)\,a_1^{(3)}}\end{aligned}

One layer back: the first weight in hidden layer 3

z1(3)=w11(3)a1(2)+w12(3)a2(2)+w13(3)a3(2)+b1(3)a1(3)=f ⁣(z1(3))\begin{aligned}z_1^{(3)}&=w_{11}^{(3)}a_1^{(2)}+w_{12}^{(3)}a_2^{(2)}+w_{13}^{(3)}a_3^{(2)}+b_1^{(3)}\\[12pt]a_1^{(3)}&=f\!\left(z_1^{(3)}\right)\end{aligned}

Follow w11(3)→z1(3)→a1(3)→z→p→Lw_{11}^{(3)}\to z_1^{(3)}\to a_1^{(3)}\to z\to p\to L.

∂L∂w11(3)=∂L∂p⋅∂p∂z⋅∂z∂a1(3)⋅∂a1(3)∂z1(3)⋅∂z1(3)∂w11(3)\frac{\partial L}{\partial w_{11}^{(3)}}=\frac{\partial L}{\partial p}\cdot\frac{\partial p}{\partial z}\cdot\frac{\partial z}{\partial a_1^{(3)}}\cdot\frac{\partial a_1^{(3)}}{\partial z_1^{(3)}}\cdot\frac{\partial z_1^{(3)}}{\partial w_{11}^{(3)}}

The three new derivatives

∂z∂a1(3)=w11(4)∂a1(3)∂z1(3)=f′ ⁣(z1(3))∂z1(3)∂w11(3)=a1(2)\begin{aligned}\frac{\partial z}{\partial a_1^{(3)}}&=w_{11}^{(4)}\\[12pt]\frac{\partial a_1^{(3)}}{\partial z_1^{(3)}}&=f'\!\left(z_1^{(3)}\right)\\[12pt]\frac{\partial z_1^{(3)}}{\partial w_{11}^{(3)}}&=a_1^{(2)}\end{aligned}

f′ ⁣(z1(3))f'\!\left(z_1^{(3)}\right) is the derivative of the activation function at this neuron's pre-activation.

Put them together

∂L∂w11(3)=p−y^p(1−p)⋅p(1−p)⋅w11(4)⋅f′ ⁣(z1(3))⋅a1(2)=(p−y^) w11(4) f′ ⁣(z1(3)) a1(2)\begin{aligned}\frac{\partial L}{\partial w_{11}^{(3)}}&=\frac{p-\hat y}{p(1-p)}\cdot p(1-p)\cdot w_{11}^{(4)}\cdot f'\!\left(z_1^{(3)}\right)\cdot a_1^{(2)}\\[12pt]&=\boxed{(p-\hat y)\,w_{11}^{(4)}\,f'\!\left(z_1^{(3)}\right)\,a_1^{(2)}}\end{aligned}

One more layer back: the first weight in hidden layer 2

Two paths from the layer-2 weight to the lossThe weight w11 in layer 2 affects z1 then a1 in layer 2. This activation feeds both neurons in layer 3, each computing its own pre-activation and activation. Their weighted contributions add at the output logit z. Sigmoid produces p, and binary cross-entropy compares p with the ground truth to produce L. Other inputs and biases are held fixed.Path 1 · G₁Path 2 · G₂× a₁⁽¹⁾f× w₁₁⁽³⁾× w₂₁⁽³⁾ff× w₁₁⁽⁴⁾× w₁₂⁽⁴⁾σBCEw₁₁⁽²⁾z₁⁽²⁾a₁⁽²⁾z₁⁽³⁾a₁⁽³⁾z₂⁽³⁾a₂⁽³⁾zpL
G₁ follows the upper path; G₂ follows the lower path. Both share the steps before and after the branch. Other inputs and biases are omitted.

Multiply the derivatives along each path, then add. G1\ G_1 and G2G_2 are the two contributions to the same weight's gradient.

Path 1 · through neuron 1

G1=∂L∂p⋅∂p∂z⋅∂z∂a1(3)⋅∂a1(3)∂z1(3)⋅∂z1(3)∂a1(2)⋅∂a1(2)∂z1(2)⋅∂z1(2)∂w11(2)=p−y^p(1−p)⋅p(1−p)⋅w11(4)⋅f′ ⁣(z1(3))⋅w11(3)⋅f′ ⁣(z1(2))⋅a1(1)=(p−y^) w11(4) f′ ⁣(z1(3)) w11(3) f′ ⁣(z1(2)) a1(1)\begin{aligned}G_1&=\frac{\partial L}{\partial p}\cdot\frac{\partial p}{\partial z}\cdot\frac{\partial z}{\partial a_1^{(3)}}\cdot\frac{\partial a_1^{(3)}}{\partial z_1^{(3)}}\\[12pt]&\quad\cdot\frac{\partial z_1^{(3)}}{\partial a_1^{(2)}}\cdot\frac{\partial a_1^{(2)}}{\partial z_1^{(2)}}\cdot\frac{\partial z_1^{(2)}}{\partial w_{11}^{(2)}}\\[16pt]&=\frac{p-\hat y}{p(1-p)}\cdot p(1-p)\cdot w_{11}^{(4)}\cdot f'\!\left(z_1^{(3)}\right)\\[8pt]&\quad\cdot w_{11}^{(3)}\cdot f'\!\left(z_1^{(2)}\right)\cdot a_1^{(1)}\\[16pt]&=(p-\hat y)\,w_{11}^{(4)}\,f'\!\left(z_1^{(3)}\right)\,w_{11}^{(3)}\,f'\!\left(z_1^{(2)}\right)\,a_1^{(1)}\end{aligned}

Path 2 · through neuron 2

G2=∂L∂p⋅∂p∂z⋅∂z∂a2(3)⋅∂a2(3)∂z2(3)⋅∂z2(3)∂a1(2)⋅∂a1(2)∂z1(2)⋅∂z1(2)∂w11(2)=p−y^p(1−p)⋅p(1−p)⋅w12(4)⋅f′ ⁣(z2(3))⋅w21(3)⋅f′ ⁣(z1(2))⋅a1(1)=(p−y^) w12(4) f′ ⁣(z2(3)) w21(3) f′ ⁣(z1(2)) a1(1)\begin{aligned}G_2&=\frac{\partial L}{\partial p}\cdot\frac{\partial p}{\partial z}\cdot\frac{\partial z}{\partial a_2^{(3)}}\cdot\frac{\partial a_2^{(3)}}{\partial z_2^{(3)}}\\[12pt]&\quad\cdot\frac{\partial z_2^{(3)}}{\partial a_1^{(2)}}\cdot\frac{\partial a_1^{(2)}}{\partial z_1^{(2)}}\cdot\frac{\partial z_1^{(2)}}{\partial w_{11}^{(2)}}\\[16pt]&=\frac{p-\hat y}{p(1-p)}\cdot p(1-p)\cdot w_{12}^{(4)}\cdot f'\!\left(z_2^{(3)}\right)\\[8pt]&\quad\cdot w_{21}^{(3)}\cdot f'\!\left(z_1^{(2)}\right)\cdot a_1^{(1)}\\[16pt]&=(p-\hat y)\,w_{12}^{(4)}\,f'\!\left(z_2^{(3)}\right)\,w_{21}^{(3)}\,f'\!\left(z_1^{(2)}\right)\,a_1^{(1)}\end{aligned}

Add both contributions

∂L∂w11(2)=G1+G2\boxed{\frac{\partial L}{\partial w_{11}^{(2)}}=G_1+G_2}