Multi-Modal Retrieval: Beyond Text

A design team searches for assets using natural language, "minimalist blue logo with a mountain", but their actual asset library is a folder of images with no useful text description, only filenames like IMG_4471.png. The current system only supports filename search, so designers waste hours a day manually scrolling through folders looking for assets they know exist somewhere. Everything covered in this tier so far assumed text-to-text retrieval. This chapter is about retrieving across a modality boundary: text queries against image content.

A standard text embedding model only knows how to embed text, and a standard image model only knows how to embed images, and critically, their two output spaces have no relationship to each other, a text embedding and an image embedding trained independently cannot be meaningfully compared even if you wanted to. Multi-modal embedding models solve this by training on paired data, images alongside their captions, so that a text description and the image it describes land close together in one single, shared embedding space. Cosine similarity between a text query's embedding and an image's embedding then works exactly the same way it did for text-to-text search in earlier chapters.

CLIP (Contrastive Language-Image Pretraining) is the best-known approach to this, trained on hundreds of millions of image-caption pairs scraped from the web, learning to pull matching image-text pairs close together in embedding space while pushing mismatched pairs apart. Once trained, a CLIP-style model can embed a text query ("minimalist blue logo with a mountain") and every image in an asset library into that same shared space, and a nearest-neighbor search, the exact same mechanism from the vector database chapter earlier in this tier, finds the images whose visual content the model judges closest in meaning to the text description.

Real multi-modal retrieval systems often go further than pure CLIP embeddings, fusing multiple signals: the CLIP embedding itself, any available text metadata (a filename, alt text, a tag someone manually applied), and sometimes OCR'd text extracted from within the image (a logo containing readable text, a screenshot of a document). Combining these, similar in spirit to the hybrid search fusion covered earlier in this tier, catches cases a single signal alone would miss.