Building a Transformer from scratch

I wanted to understand attention, not just call nn.Transformer. So I wrote an encoder-decoder model for English-to-Italian translation: embeddings, multi-head attention, positional encoding, and the training loop.

The unglamorous parts taught me the most. Padding masks, causal masks, tokenizer edge cases, and checkpointing all showed up before the loss looked healthy. Once those were right, the model started producing actual translations instead of noise.

If you are learning transformers, implement one. The paper makes more sense after you have fought with the shapes.