Case Study: Visual Search for a Marketplace

A shopper sees a lamp in a friend's house, takes a photo, and wants to find something like it for sale. A marketplace with 100 million listings must answer in a fraction of a second. Visual search is a direct use of the embedding and nearest-neighbor ideas from the ML systems module, and its main design problem is a practical one: fitting a huge index in memory.

Imagine a shop assistant who does not read labels. They look at the lamp in your photo and point to the shelves with lamps that look similar. The assistant has a trained eye: they have seen so many lamps that they know which differences matter. An image encoder is that trained eye, and it turns each picture into a vector, so that similar-looking items are close.

• -Embedding: an image encoder, trained so that pictures of the same or similar products land near each other, turns every listing photo into a vector, offline. A model that links images and text, like the contrastive models from the diffusion and multimodal module, can also embed words into the same space.
• -Index: the vectors go into an approximate nearest-neighbor index, as in the retrieval chapter.
• -Query: the shopper's photo is embedded on the fly, with some cropping or detection of the main object, and the index returns candidates.
• -Filters: category, price, location, and availability are applied during the search, so results are things that can actually be bought.
• -Re-ranking: a second model combines visual similarity with listing quality, seller trust, price, and the user's history.
• -Refinement: "like this, but in blue" combines the image vector with a text vector, or applies a filter.

The index has to be fast, so it lives in memory. How big is it? 100 million vectors, each with 512 numbers. The code computes the size for four storage formats, and the number of 64 GB machines needed if only 60% of each machine's memory can hold the index. Press Run and read the last two columns.

As 32-bit floats the vectors take 204.8 GB, which needs 6 machines of 64 GB at 60% usable. As 16-bit floats they take 102.4 GB and 3 machines. As 8-bit integers they take 51.2 GB and 2 machines. With product quantization, which replaces each vector with a short code of 64 bytes, the index shrinks to 6.4 GB and fits on one machine. Each step down loses some recall, so the right choice is the most compressed format whose recall on your own test queries is still good enough, often followed by a re-score of the top results using the full vectors kept on disk.

• -New listings appear constantly and sold items must disappear. Many ANN indexes support adding vectors incrementally, and removal is usually a delete marker plus a periodic rebuild.
• -Embeddings should come from a versioned encoder. A new encoder means re-embedding everything and swapping the index, so plan for a blue-green index switch.
• -Availability and price change faster than visuals, so keep them as filters on fresh data, not baked into the vector.

• -Offline: a set of query photos with human-labeled matches, scored by recall at k and NDCG, split by category, since lamps and shoes behave differently.
• -Online: click-through and add-to-cart on results, and the share of searches that end in a purchase, with an A/B test.
• -Quality slices: poor photos, cluttered backgrounds, and items with multiple objects.

• -Counterfeits and lookalikes: visual similarity cannot tell a genuine product from a copy. Add seller and listing signals.
• -Safety and copyright: filter disallowed items, and handle images of people with care.
• -Privacy: users upload photos that may contain faces or locations. Decide what to store, and for how long.
• -Bias: encoders can work worse for some products or cultures. Evaluate across segments.