What Are Embeddings, Really?

Tokenization turns text into discrete pieces. Embeddings turn each piece, or a whole chunk of text, into something entirely different: a list of numbers, a vector, that acts as coordinates in a high-dimensional space where distance means something. Two pieces of text with similar meaning end up as vectors that sit close together. Two pieces of text about unrelated topics end up far apart. This single idea, meaning as geometry, is the foundation underneath semantic search, recommendation systems, and every retrieval step you will build in the RAG tier right after this one.

A traditional keyword search engine matches exact substrings: search "puppy" and it will never find a document that only says "dog." An embedding model instead maps both words to nearby points in space, because it learned, from enormous amounts of text, that they tend to appear in similar contexts. Real embedding models produce vectors with hundreds or thousands of dimensions, far too many to visualize directly, but the underlying arithmetic is exactly the same regardless of dimension count, which is why a small, hand-built example is enough to make the idea concrete.

The standard way to measure how similar two embedding vectors are is cosine similarity: the cosine of the angle between them, ranging from -1 (pointing in opposite directions) through 0 (unrelated, at a right angle) to 1 (pointing in exactly the same direction). It cares about direction, not magnitude, which matters because it means a short document and a long document about the same topic can still land as highly similar, as long as they point the same way in meaning-space.

This prints cos(king, queen) = 0.861, cos(king, man) = 0.992, and cos(king, banana) = 0.203. Notice king is actually more similar to man than to queen in this toy example, which is not a mistake: these hand-built vectors share a strong "male" component along with the "royalty" one, and a real embedding blends every semantic axis it learned into a single vector, it does not neatly separate "gender" from "title" for you. The last line is the famous embedding-arithmetic demonstration: subtracting the "man" vector and adding the "woman" vector shifts "king" almost exactly onto "queen" (a similarity of 1.0 here, because this toy example is deliberately constructed to make the geometry obvious; real embeddings land close but rarely land exactly on top of the target).

• -Semantic search: embed a query and every candidate document, then rank documents by cosine similarity to the query, finding conceptually related results even with zero shared keywords. This is the retrieval half of RAG, covered in full in the next tier.
• -Deduplication and clustering: near-duplicate content lands as near-identical vectors, and topic clusters emerge naturally as groups of nearby points.
• -Recommendation: "more like this" is, underneath, "find the nearest neighbors of this vector."