Embeddings & Two-Tower Retrieval

A video platform has hundreds of millions of videos. When you open the app, it cannot run a large neural network on every video to decide what to show you. Somehow it must go from a huge catalog to a short list in a few milliseconds. The standard tool for that first narrowing is a two-tower model with an approximate nearest-neighbor index.

Imagine every person and every book placed on a giant map, where nearby points share tastes. To recommend books to a reader, you find the nearest book points to their point. You never read every book for every visitor. The books were placed on the map once, ahead of time. A two-tower model learns that map. One tower places users, the other places items.

• -The user tower is a neural network that turns user features (history, location, device) into a vector.
• -The item tower is a second network that turns item features (title, category, creator) into a vector of the same size.
• -The score for a pair is the dot product of the two vectors. Higher means a better match.
• -Training uses examples where a user interacted with an item as positives. Other items in the same training batch act as negatives, a common and cheap trick called in-batch negatives. Choosing harder negatives is a topic of its own.

The two towers never look at each other's inputs until the final dot product. That separation is the point. It means every item vector can be computed in advance, offline, and stored. At request time only the user's vector is computed, and the question becomes a nearest-neighbor search over precomputed vectors. A model that read each user and item together, which can be more accurate, would have to run once per pair, far too many pairs to score.

Finding the exact nearest vectors means comparing the query with every item, which is too slow for millions of items. Approximate nearest-neighbor (ANN) indexes give up a little accuracy for a lot of speed. Well-known families include graph indexes such as HNSW and clustering-based indexes. The idea in each is to examine only a small part of the catalog that is likely to contain the best matches.

The code below builds a toy version using random hyperplanes: each hyperplane splits the space in two, and items that land on the same side of every plane share a bucket. A query looks only at items in its own bucket. Using more tables of buckets means scanning more items and finding more of the true nearest neighbors. Press Run, and read the table from the top: each row spends more work and buys more recall.

The first row scans about 0.7% of the catalog and finds under 8% of the true top ten. At 16.6% scanned, recall is 81.3%, and at 34.7% it reaches 95.7%. That is the trade every ANN index makes: recall against work. This toy is a hard case, since random vectors have no clusters. Real embeddings are clustered, and production indexes such as HNSW get much higher recall for far less scanning. The shape of the trade-off is the same.

Eugene Yan's survey of industry discovery systems describes the same two stages everywhere. Retrieval is fast and coarse. It cuts millions of items to hundreds, roughly a 99.99% reduction, usually with an ANN search over embeddings. Ranking is slower and more precise, and it scores only those few hundred with richer features and a larger model. The YouTube paper from 2016 describes the same split: a candidate generation network followed by a ranking network that looks at a few hundred videos.

• -**Retrieval**: Optimize for recall. It must keep the good items in the shortlist, quickly and cheaply, so it uses simple scoring such as a dot product.
• -**Ranking**: Optimize for precision. It orders the shortlist using many features and a heavier model, since only hundreds of items need scoring.

This makes retrieval recall a ceiling on the whole system. If the right item is not in the shortlist, no ranker can bring it back. It is the first number to check when quality is poor.