Architectures at a Glance: A Preview of Transformers

Everything so far in this module described the generic building blocks: neurons, layers, activations, forward passes, backpropagation. But real networks are not just an undifferentiated stack of identical layers. They are arranged into specific architectures, shaped around the structure of the data they are meant to process.

• -Convolutional networks (CNNs) are shaped around images. They apply small, reusable filters across an image's grid of pixels, well suited to the fact that a pattern like an edge or a texture can appear anywhere in a photo.
• -Recurrent networks (RNNs) were the traditional shape for sequential data like text or audio, processing one element at a time while carrying forward a running memory of everything seen so far.
• -Transformers are the architecture that now dominates language, and increasingly other domains too. Instead of processing a sequence one step at a time like an RNN, they let every position in a sequence directly weigh how relevant every other position is, all at once, via a mechanism called self-attention.

Nothing in this module ever mentioned where a network's starting weights actually come from in practice, beyond "small random numbers." Training a large network entirely from scratch on your own data is expensive and often unnecessary. Transfer learning instead starts from a network that has already been trained on a huge, general dataset, then adapts it to a new, more specific task.

There are two common ways to adapt a pretrained network. Freezing keeps most of the pretrained layers exactly as they are and only trains a small new layer added on top, treating the frozen layers as a fixed feature extractor. Fine-tuning goes further and continues training some or all of the pretrained layers too, usually with a much smaller learning rate than you would use from scratch, so the existing weights shift gently instead of being overwritten.

The transformer architecture, introduced in a landmark 2017 paper, is specifically what makes modern large language models possible. It is why Phase 2 of this course exists as a distinct phase rather than being folded into this one. Understanding that it is "just" a specific architectural choice, neurons, layers, and backpropagation underneath, arranged in a particular shape well suited to language, should make it feel far less like magic once Phase 2 covers it in depth, including its own use of the exact dot-product operation from the Math module to compute attention scores.