🧬LLM Batch API
Also known as Batch API (LLM), Batch Processing (LLM), Message Batches
Send lots of model requests at once. The answers come back later and cost less.
Think of it like
Think of it like dropping off a big pile of laundry. It costs less, and you pick it up later.
Example
Jack needs short summaries of 40,000 old product reviews. He sends them as one batch overnight and pays about half of what single calls would cost.
Why it matters
Big jobs that can wait cost about half as much. They also leave your rate limits free for the live app.
Definition
Example code
json
{"custom_id": "review-1", "params": {"model": "model-name", "max_tokens": 100, "messages": [{"role": "user", "content": "Summarize: ..."}]}}Common mistake
Key takeaways
- 1Many at once. One file in.
- 2Answers later. Often within a day.
- 3Lower price. Often about half.