Skip to content
BENIBIX Learn

Multimodal Embedding

AI🎨 MultimodalVision and Language

Also known as Image-Text Embedding, Joint Embedding, Cross-Modal Embedding

Numbers that put pictures and words in one shared space. Then they can be compared.

Think of it like

It is like one big map. A photo of a dog and the word dog get the same pin.

Example

Lucy types 'red shoes on grass' into her photo app. It finds the right picture among 8,000, though no one ever tagged it.

Why it matters

It lets one search work across words and pictures. You can find an image with a sentence, or a sentence with an image.

Definition

A multimodal embedding turns different kinds of input, such as text and images, into lists of numbers in one shared space. Things that mean the same land close together, whatever form they come in. A photo of a beach and the words 'a sunny beach' end up near each other. This lets you search photos by typing words, or find text that matches a picture. Models trained on huge sets of image and caption pairs learn this shared space.
Right: the photo and the words land on the same spot, so one can find the other.

Common mistake

Comparing numbers made by two different models. Only items from the same model share one space.

Key takeaways

  1. 1
    One shared space. Text and images sit together.
  2. 2
    Close means alike. Same meaning, nearby points.
  3. 3
    Same model only. Do not mix two models' numbers.

Compare