Skip to content
BENIBIX Learn

LLM Batch API

Development🧬 AI EngineeringLLM APIs

Also known as Batch API (LLM), Batch Processing (LLM), Message Batches

Send lots of model requests at once. The answers come back later and cost less.

Think of it like

Think of it like dropping off a big pile of laundry. It costs less, and you pick it up later.

Example

Jack needs short summaries of 40,000 old product reviews. He sends them as one batch overnight and pays about half of what single calls would cost.

Why it matters

Big jobs that can wait cost about half as much. They also leave your rate limits free for the live app.

Definition

A batch API lets you send lots of model requests in one go and get the answers later, for less. You send a file of requests, each with its own id. The model maker runs them when it has spare room. It sends back a file of results, most often within a day. The price is often about half that of a normal call. It suits big jobs where no one is waiting, such as tagging a whole archive.

Example code

json
{"custom_id": "review-1", "params": {"model": "model-name", "max_tokens": 100, "messages": [{"role": "user", "content": "Summarize: ..."}]}}
One line of a batch file. Each request carries its own id so the results can be matched up.
A file of requests goes in at once. A file of answers comes back later, at a lower price.

Common mistake

Using it for anything a user is waiting on. Results can take hours, so live features need normal calls.

Key takeaways

  1. 1
    Many at once. One file in.
  2. 2
    Answers later. Often within a day.
  3. 3
    Lower price. Often about half.