Skip to content
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 4 min

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · NAACL 2019

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

How hiding 15% of the words taught machines to read in both directions. Masked language modeling, the 80/10/10 rule and fine-tuning, with a mask generator you can edit.

Read the original paper

AI & ML · Yellow Belt · Pro · 4 min read

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Read this sentence with one word hidden: "The river ___ was muddy." You can guess "bank" because you use the words on both sides. "The river" before it tells you the setting, and "was muddy" after it confirms it. Now imagine you could only read left to right and had to guess each word before seeing what follows. That is how most language models were trained in 2018.

BERT, from Google researchers, changed that. It is a model of language that is trained to fill in hidden words using context from both directions at once. After this pre-training, one model can be adapted to many tasks with only a little extra work. At the time it set new records on eleven language-understanding tasks.

BERT is a transformer encoder pre-trained by hiding about 15% of the words in huge amounts of plain text and predicting them from both sides. The result is a general-purpose language understander that you fine-tune for each task by adding a small output layer.

Why both directions matter

Earlier language models such as OpenAI's GPT were unidirectional. Each word could only look at the words before it. That is natural if you want to generate text one word at a time, but it limits how well the model can understand a sentence. Understanding tasks such as question answering depend on context from both sides.

Training a bidirectional model sounds easy, but it breaks the usual objective. If a word can see the words after it, then predicting the next word is trivial, because the answer is right there. BERT's trick is to change the question.

A left-to-right model predicts each word from the words before it. BERT hides some words and predicts them from the words on both sides, which is much better at understanding a sentence.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Masked language modeling
  • Pre-train once, fine-tune many times

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.