Self-Supervised Learning: A Preview of What's Next
You'll learn to
- -Understand self-supervised learning as a way to generate labels automatically from unlabeled data
- -Turn one sentence into several training examples, and see why the supply is almost unlimited
- -See why this technique is the key that unlocked large language model training
- -Evaluate a claim of needing no labeled data at all
Tip: the highlighter is on - just select any text below to mark it. Use the highlighter button up top to change color or turn it off. Highlights are saved on this device only. Log in to save them to your account instead.
Self-supervised learning is the technique that made today's large language models possible, and it deserves its own introduction here before Phase 2 goes deep on how LLMs actually work. It solves supervised learning's biggest bottleneck, expensive human labels, with a clever trick: generate the labels automatically from the data itself.
An Analogy: Fill-in-the-Blank Practice
Imagine learning a language by reading books with some words blanked out and filling them in. The answers are on the next page, so no teacher is needed: the book itself is the answer key. The more books you can read, the better you get. Self-supervised learning is that exercise, run on a computer across billions of pages.
The Trick: Hide Part of the Data, Predict It
Take a sentence, hide a word, and ask the model to predict the missing word from the surrounding context. The "label" is just the word that was already there. No human had to tag anything. Do this across billions of sentences from the internet, and a model can learn an enormous amount about language, facts, and reasoning patterns without a single human-annotated example.
Here is how much training data a single sentence provides. Take the six-word sentence the cat sat on the mat. Ask the model to predict each word from the words before it:
Five examples from six words, and no human wrote a single label. In general a sentence of n words gives n - 1 examples, so a text collection of a trillion words gives close to a trillion examples. That is why the supply is almost unlimited, and why human labeling, which costs real money per example, could never keep up.
- -Supervised learning relies on humans providing labels, which is expensive and limited in scale.
- -Self-supervised learning has the data provide its own labels by hiding and predicting parts of itself, at nearly unlimited scale, since almost any text, image, or audio can be used.
Two flavors are worth knowing by name. In masked prediction, a word in the middle is hidden and the model fills it in from both sides, the approach of BERT-style models. In next-token prediction, the model predicts what comes next, the approach of GPT-style models and most of the language models in this course. The same idea works for images (hide a patch and predict it) and audio.
This is technically still a form of supervised learning at the mechanical level. There is still a "correct answer" being predicted. The innovation is entirely in where the label comes from, which is why it gets its own name: self-supervised.
This single idea is what allowed models to be trained on a meaningful fraction of all publicly available text, rather than being limited to the comparatively tiny amount of data humans could afford to hand-label. It is the foundation Phase 2 builds directly on top of.
That also tells you how to read a claim such as our model needed no labeled data at all. It may be true that no humans labeled anything for the main training. But the labels still existed, hidden inside the data. And models usually need a small amount of labeled data later, for example to teach them to follow instructions or to be tested. So ask: no human labels for which stage?
Someone claims their new model needed "no labeled data at all" to train. Based on what self-supervised learning actually does, how would you evaluate that claim?
"I would take the claim at face value, since the model clearly was not trained on human-tagged examples."
"I would push back gently on the framing rather than the substance. Self-supervised learning still trains on a prediction target, a hidden word or masked region, it is just that the label comes from the data itself rather than a human annotator, so it is technically still a form of supervised learning at the mechanical level. The genuinely new part is where the label comes from, not the absence of a label entirely. I would want to know specifically what was being predicted and from what context, since that is what determines what the model actually learned."
What problem does self-supervised learning solve?
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!