Skip to content
GenAI Learn/Diffusion Models & Multimodal Generation
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Contrastive Embeddings: How CLIP Connects Images and Text

5 min read

You'll learn to

  • -Explain a joint embedding space with the matchmaking-party picture: why an image and a sentence can be compared at all
  • -Read the CLIP similarity table of a batch and say what the diagonal means
  • -Define cross-entropy in plain words and compute why mismatched pairs give a much higher loss
  • -Say what the scale (temperature) does and why the loss is computed in both directions

The GenAI Lab's Multi-Modal Retriever level searches a folder of images using a sentence. That sounds impossible. A picture is a grid of pixels and a sentence is a string of words, and the Math module only taught you to compare two vectors of the same kind. How can the two be compared at all? The answer is CLIP, and it comes from a specific training idea called contrastive learning.

An Analogy: A Matchmaking Party

Imagine a ballroom where every guest is either a photo or a caption. At the start everyone stands in random places. The organizer knows which photo belongs with which caption, and after each round moves every photo a little closer to its own caption and a little further from all the other captions. After millions of rounds, photos stand next to captions that describe them. To find the picture for "a red bicycle", you place that sentence in the room and look at which photos are standing nearest.

  • -The ballroom is the joint embedding space: one shared set of coordinates for both kinds of guest.
  • -A guest's position is its embedding: a list of numbers that captures what it means.
  • -There are two doors into the ballroom. The image encoder decides where a photo stands and the text encoder decides where a caption stands.
  • -A photo and its own caption are a matched pair.
  • -Standing close together means a high dot product, the similarity measure from the Math module.

Two Terms You Need: Logits and Cross-Entropy

Logits are raw scores before softmax turns them into probabilities. Cross-entropy is a way to grade a probability: it is the negative log of the probability that was given to the right answer. If the model gave the right answer 0.9, the loss is -log 0.9 = 0.105, small. If it gave the right answer only 0.1, the loss is 2.30, large. Confident and right is cheap, confident and wrong is expensive. This is the same loss you met in the Math module.

The Contrastive Training Objective

Take a batch of N images, each with its own caption. Embed all of them and compute every image's dot product with every caption. That makes an N by N table of similarity scores. Row i holds image i's scores against all N captions, and exactly one of them is correct: caption i, on the diagonal.

So each row is a multiple-choice question, "which of these N captions belongs to this image?", and the correct label is simply the diagonal position. Grade it with cross-entropy. Then do the same by columns, "which of these N images belongs to this caption?". Averaging the two directions stops the space from becoming lopsided in favor of just one of them.

CLIP Contrastive Loss
L = ½·[CE(scale·I·Tᵀ, diag) + CE(scale·T·Iᵀ, diag)]
I and T are the batch's normalized image and text embeddings. scale is a learned number that makes the scores more or less decisive before softmax. The loss is computed in both directions, image-to-text and text-to-image, and averaged.

Try a tiny batch of two pairs, say a cat and a dog, with simple two-number embeddings. With the captions matched, the loss is 0.31. Swap the two captions and the loss jumps to 1.31, because each picture now sits next to the wrong caption. Raise the scale to 10 and the contrast is stark: about 0 for the right pairs and 10 for the swapped ones. A high scale makes the model much more decisive, so wrong pairs are punished harder.

The CLIP contrastive loss, from scratch

What This Buys You

  • -No manual labeling beyond the pairing itself. Which caption came with which image is exactly the metadata a normal image-and-caption dataset already has.
  • -Once trained, a plain dot product between an image embedding and a text embedding is a meaningful similarity score. That is what makes searching images with a text description possible, and also zero-shot classification: compare a photo against captions such as "a photo of a cat" and "a photo of a dog" and pick the closest.
  • -The same idea trains the embedding models behind text-only search. The vector databases from the RAG module rely on embeddings trained with a close cousin of this objective. CLIP simply runs it across two different encoders instead of one.

AI Lab's Level 71 (Contrastive Embeddings, CLIP-Style) implements this in numpy across a full batch at once, using matrix multiplication for the whole similarity table instead of a nested loop over pairs.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement it from scratch: Level 71 (Contrastive Embeddings / CLIP-Style Loss) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!