Skip to content
GenAI Learn/How Machines Learn
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Self-Supervised Learning: A Preview of What's Next

5 min read

You'll learn to

  • -Understand self-supervised learning as a way to generate labels automatically from unlabeled data
  • -Turn one sentence into several training examples, and see why the supply is almost unlimited
  • -See why this technique is the key that unlocked large language model training
  • -Evaluate a claim of needing no labeled data at all

Tip: the highlighter is on - just select any text below to mark it. Use the highlighter button up top to change color or turn it off. Highlights are saved on this device only. Log in to save them to your account instead.

Self-supervised learning is the technique that made today's large language models possible, and it deserves its own introduction here before Phase 2 goes deep on how LLMs actually work. It solves supervised learning's biggest bottleneck, expensive human labels, with a clever trick: generate the labels automatically from the data itself.

An Analogy: Fill-in-the-Blank Practice

Imagine learning a language by reading books with some words blanked out and filling them in. The answers are on the next page, so no teacher is needed: the book itself is the answer key. The more books you can read, the better you get. Self-supervised learning is that exercise, run on a computer across billions of pages.

The Trick: Hide Part of the Data, Predict It

Take a sentence, hide a word, and ask the model to predict the missing word from the surrounding context. The "label" is just the word that was already there. No human had to tag anything. Do this across billions of sentences from the internet, and a model can learn an enormous amount about language, facts, and reasoning patterns without a single human-annotated example.

Here is how much training data a single sentence provides. Take the six-word sentence the cat sat on the mat. Ask the model to predict each word from the words before it:

One sentence of 6 words gives 5 training examples, each with its label already in the text

Five examples from six words, and no human wrote a single label. In general a sentence of n words gives n - 1 examples, so a text collection of a trillion words gives close to a trillion examples. That is why the supply is almost unlimited, and why human labeling, which costs real money per example, could never keep up.

  • -Supervised learning relies on humans providing labels, which is expensive and limited in scale.
  • -Self-supervised learning has the data provide its own labels by hiding and predicting parts of itself, at nearly unlimited scale, since almost any text, image, or audio can be used.

Two flavors are worth knowing by name. In masked prediction, a word in the middle is hidden and the model fills it in from both sides, the approach of BERT-style models. In next-token prediction, the model predicts what comes next, the approach of GPT-style models and most of the language models in this course. The same idea works for images (hide a patch and predict it) and audio.

This is technically still a form of supervised learning at the mechanical level. There is still a "correct answer" being predicted. The innovation is entirely in where the label comes from, which is why it gets its own name: self-supervised.

This single idea is what allowed models to be trained on a meaningful fraction of all publicly available text, rather than being limited to the comparatively tiny amount of data humans could afford to hand-label. It is the foundation Phase 2 builds directly on top of.

That also tells you how to read a claim such as our model needed no labeled data at all. It may be true that no humans labeled anything for the main training. But the labels still existed, hidden inside the data. And models usually need a small amount of labeled data later, for example to teach them to follow instructions or to be tested. So ask: no human labels for which stage?

Interview Signal

Someone claims their new model needed "no labeled data at all" to train. Based on what self-supervised learning actually does, how would you evaluate that claim?

Weak Answer

"I would take the claim at face value, since the model clearly was not trained on human-tagged examples."

Strong Answer

"I would push back gently on the framing rather than the substance. Self-supervised learning still trains on a prediction target, a hidden word or masked region, it is just that the label comes from the data itself rather than a human annotator, so it is technically still a form of supervised learning at the mechanical level. The genuinely new part is where the label comes from, not the absence of a label entirely. I would want to know specifically what was being predicted and from what context, since that is what determines what the model actually learned."

Check Yourself1 / 5

What problem does self-supervised learning solve?

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!