Skip to content
BENIBIX Learn

Contrastive Language-Image Pre-Training (CLIP)

Also known as CLIP, OpenAI CLIP

An OpenAI model. It finds the words that best fit a picture.

Think of it like

It is like a game. You match each photo to the right label.

Example

Emma has 5,000 photos with no names. She asks CLIP to find 'a dog on a beach', and it ranks the photos that fit best.

Why it matters

It taught models to understand pictures through plain words. Photo search and many picture makers use ideas from it.

Definition

CLIP is a model from OpenAI that links pictures and words. It was trained on a huge set of pictures paired with their captions. It learned to put a picture and the words that fit it close together. So you can show it a photo and some captions, and it picks the one that fits best. It can do this for new labels with no extra training. Its code and weights are open, under the MIT license.
CLIP takes a picture and some captions in, and gives the best match out.

Common mistake

Thinking CLIP makes pictures. It only matches and scores them; other models do the drawing.

Key takeaways

  1. 1
    Pictures and words. Placed in one shared space.
  2. 2
    Pick the match. It scores each caption.
  3. 3
    No retraining. It works on new labels.

Learn it properly