LLM Gateway vs API Gateway
People often think an LLM gateway is just an API gateway with a new name. Both sit in front of services and deal with keys, limits and logs. An API gateway guards your own services from outside callers. An LLM gateway sits in front of outside model makers for your own apps. It counts tokens, tracks the cost of each model and can switch model makers.
LLM Gateway
One place that all the AI calls in a firm go through.
An LLM gateway is a service that sits between your apps and the model makers, so every model call goes through one place. Apps call the gateway, and it sends each request on to the right model maker. It holds the API keys and sets limits on spend and on calls. It logs every call and its cost, and it can switch to another model maker when one fails. Teams use one when they have many apps or many models.
API Gateway
One front door that sends each request to the right service.
An API gateway is a single front door that sits in front of many back-end services. Every request from an app goes to the gateway first. It checks who is calling, applies rate limits, and then sends the request on to the right service. It can also combine answers from several services into one reply. Clients only need to know one address.
In one line
- LLM Gateway
- One place that all the AI calls in a firm go through.
- API Gateway
- One front door that sends each request to the right service.