Skip to content
BENIBIX Learn

Vision Language Model (VLM)

AI🎨 MultimodalVision and Language

Also known as VLM, Vision-Language Model, Visual Language Model

An AI that can look at a photo. Then it tells you about it in words.

Think of it like

It is like a friend who can look at a photo with you. Then you can ask them what is in it.

Example

Emma sends a photo of a broken bike chain to a chat app and asks what is wrong. The model sees that the chain has come off the back gear and tells her how to put it back.

Why it matters

Much of the world is pictures, charts and screens, not text. A model that can see can help with all of them.

Definition

A vision language model is an AI model that can take in pictures as well as text. It answers in text. First, an image encoder turns the picture into tokens, small bits the model can read like words. The model then reads those tokens along with your question. So it can describe a photo, read a chart, or answer questions about a screen. Most big chat models today can do this.
The photo turns into tokens. They join the question, and the model writes the answer.

Common mistake

Trusting it to read small text or count many things exactly. It can miss fine detail, so check numbers it reads from a picture.

Key takeaways

  1. 1
    Sees and reads. Pictures and text.
  2. 2
    Image to tokens. Then read like words.
  3. 3
    Answers in text. Describe, read, explain.

Compare