Before 2013, the standard way to feed a word into a machine learning model was a one-hot vector: a list as long as your vocabulary, all zeros except a single 1 marking which word it is. It works, but it throws away everything useful about word representations. Under one-hot encoding, "cat" and "dog" are exactly as similar to each other as "cat" and "spreadsheet", which is to say, not similar at all, because every pair of distinct words is equally far apart.
Neural language models had already shown a fix: learn a dense vector for each word instead, trained so that words used in similar contexts end up with similar vectors. The problem was cost. Existing neural language models had a full nonlinear hidden layer and a softmax over the entire vocabulary, which made them far too slow to train on the billions of words of text that were increasingly available. Tomas Mikolov and colleagues at Google asked a blunt question: how much of that architecture can we throw away and still get good word vectors, if what we actually want is the vectors themselves, not a language model?
Word2vec strips a neural language model down to the simplest architecture that still learns useful word vectors: no nonlinear hidden layer at all, just a projection into a shared vector space. This makes it cheap enough to train on billions of words, and the resulting vectors capture semantic and syntactic relationships well enough that simple vector arithmetic, king minus man plus woman, lands close to queen.
The idea underneath: you know a word by its company
The whole approach rests on the distributional hypothesis, a decades-old idea from linguistics: words that occur in similar contexts tend to have similar meanings. "Coffee" and "tea" show up near words like "cup", "drink", and "morning" far more often than either shows up near "quantum" or "spreadsheet". If you train a model to predict a word from its neighbors, or its neighbors from it, purely as a side effect of getting good at that prediction task, words with similar usage patterns are pushed toward similar vectors.
Nobody tells the model what "similar" should mean. Similarity emerges entirely from the prediction objective: two words end up with nearby vectors only because doing so made the model better at predicting context, which is a much cheaper signal to get right at scale than any hand-built list of synonyms.
Two architectures, one shared idea
The paper proposes two log-linear models, deliberately simple enough to train fast:
Continuous Bag-of-Words (CBOW): given the words surrounding a target word, average their vectors and predict the target word. Fast to train, and tends to work slightly better for frequent words, since it is effectively averaging over many training examples per prediction.
Skip-gram: the mirror image. Given a single target word, predict each of the surrounding context words independently. Slower per epoch, but tends to do better with less data and with rare words, since every word gets to directly influence several separate training examples instead of being blended away inside an average.
CBOW predicts the center word from a bag of its neighbors. Skip-gram runs the same idea in reverse: predict each neighbor from the center word.
Both models drop the nonlinear hidden layer entirely. There is no tanh, no sigmoid, just a linear projection into a shared embedding space followed by a softmax-style prediction. That single simplification is most of why word2vec is orders of magnitude cheaper to train than the neural language models that came just a few years before it.
What the model is actually optimizing
For skip-gram, training maximizes the average log probability of context words given the center word:
where is the number of training words and is the size of the context window. The basic probability is a softmax over the dot product of the two words' vectors, which is exactly the expensive part: computing it exactly requires summing over the entire vocabulary for every single training example.
A plain softmax over a vocabulary of hundreds of thousands of words, computed for every training example, is exactly the cost this paper is trying to escape. The original paper leans on hierarchical softmax to approximate it in time logarithmic in vocabulary size instead of linear; the widely used negative sampling trick, replacing the full softmax with a handful of easy binary classification tasks, came from Mikolov's own follow-up paper later the same year and is what most modern word2vec implementations actually use.
The result that made everyone pay attention: vector arithmetic
The paper's most famous demonstration is not a benchmark score, it is an analogy task solved almost by accident. Take the vector for "king", subtract the vector for "man", add the vector for "woman", and search for the nearest vector to the result. It lands closest to "queen", reliably, across many such analogy pairs (country and capital city, verb tense, plural forms).
The offset from "man" to "king" is nearly the same direction and distance as the offset from "woman" to "queen". Relationships between words become simple vector arithmetic.
This was not something the training objective explicitly asked for. It is a side effect of a linear model: because there is no nonlinearity distorting the vector space, consistent relationships between word pairs tend to show up as consistent, nearly parallel offset vectors. That property quietly disappeared from later, more nonlinear architectures, and is part of why word2vec vectors are still specifically useful for tasks that lean on this kind of linear structure.
Why this paper still matters
Word2vec did not just produce a useful tool, it changed how the entire field thought about representing words. Within a couple of years, GloVe offered an alternative trained directly on global co-occurrence statistics, and fastText extended the idea to subword pieces so it could handle misspellings and rare words gracefully. Every embedding layer in every modern language model, including the input embeddings that feed into the Transformer covered elsewhere on this site, is a direct descendant of the idea this paper popularized: represent a discrete token as a dense, learned vector, trained purely from how it is used.
It is also a useful lesson in research taste. The paper's central bet, that a much simpler, much cheaper model trained on much more data would beat a fancier model trained on less, predates and foreshadows one of the recurring themes of the deep learning era: scale and simplicity, applied together, tend to beat cleverness applied to a smaller budget.
Key takeaways: CBOW predicts a word from its context, skip-gram predicts context from a word, and both work because dropping the nonlinear hidden layer makes training cheap enough for web-scale corpora. The resulting vector space captures relationships between words as consistent, nearly linear offsets, which is what makes vector arithmetic like king minus man plus woman work at all.
