Transformer

Encoder Architecture

What are we training?

Transformer encoder sentiment-classification architectureToken embeddings with positional information flow through a Transformer encoder block. The resulting features enter a compact MLP classifier that produces a vector of class probabilities.InputsInputEmbeddingPositional Encoding+Encoder BlockMulti-HeadSelf-AttentionAdd &NormFeedForwardAdd &NormResidual pathResidual pathFeaturesMLPClassifierProbabilitiesp₁p₂p₃⋮pₙ

Hyperparameters

Embedding dimension
d_model = 256
Attention heads
8
Feed-forward network
256 → 1024 → 256
Classifier MLP
256 → 128 → n

Inside One Self-Attention Head

Inside one self-attention headThe embeddings for Rishav, loves, and football are projected into query, key, and value matrices for the first of eight parallel attention heads. Query and key produce a three-by-three attention matrix. Multiplying those weights by value produces one contextual vector for each token.The same computation runs in parallel:H2H3H4H5H6H7H8Input embeddings XRishavlovesfootball3 × 256Head 1 of 8WQ⁽¹⁾256 × 32WK⁽¹⁾256 × 32WV⁽¹⁾256 × 32Q⁽¹⁾ = XWQ⁽¹⁾3 × 32K⁽¹⁾ = XWK⁽¹⁾3 × 32V⁽¹⁾ = XWV⁽¹⁾3 × 32Attention matrix A⁽¹⁾each row sums to 10.550.350.100.300.200.500.100.250.65A⁽¹⁾ = softmax(Q⁽¹⁾K⁽¹⁾ᵀ / √32)3 × 3O⁽¹⁾ = A⁽¹⁾V⁽¹⁾Context-aware embeddings3 × 32Rishavlovesfootball

What Happens After We Calculate Every Head?

Combining the outputs of all attention headsEight three-by-thirty-two contextual matrices are concatenated into a three-by-two-hundred-fifty-six matrix. The result passes through the output projection, is added to the original input through a residual connection, and then enters layer normalization.Eight contextual outputs—one from each attention headO⁽1⁾3 × 32O⁽2⁾3 × 32O⁽3⁾3 × 32O⁽4⁾3 × 32O⁽5⁾3 × 32O⁽6⁾3 × 32O⁽7⁾3 × 32O⁽8⁾3 × 32Concatenate O⁽¹⁾ … O⁽⁸⁾3 × (8 · 32) = 3 × 256Output projection Wᴼ(3 × 256)(256 × 256)MHA output3 × 256Original X3 × 256Residual connection+LayerNormAdd & NormFFN input3 × 256