Read this sentence with one word hidden: "The river ___ was muddy." You can guess "bank" because you use the words on both sides. "The river" before it tells you the setting, and "was muddy" after it confirms it. Now imagine you could only read left to right and had to guess each word before seeing what follows. That is how most language models were trained in 2018.
BERT, from Google researchers, changed that. It is a model of language that is trained to fill in hidden words using context from both directions at once. After this pre-training, one model can be adapted to many tasks with only a little extra work. At the time it set new records on eleven language-understanding tasks.
BERT is a transformer encoder pre-trained by hiding about 15% of the words in huge amounts of plain text and predicting them from both sides. The result is a general-purpose language understander that you fine-tune for each task by adding a small output layer.
Why both directions matter
Earlier language models such as OpenAI's GPT were unidirectional. Each word could only look at the words before it. That is natural if you want to generate text one word at a time, but it limits how well the model can understand a sentence. Understanding tasks such as question answering depend on context from both sides.
Training a bidirectional model sounds easy, but it breaks the usual objective. If a word can see the words after it, then predicting the next word is trivial, because the answer is right there. BERT's trick is to change the question.
