Model Routing vs Model Fallback
People think model routing and model fallback are the same switch between models. Routing picks a model before the call is sent, based on how hard the request looks. Fallback waits for a call to fail, then sends it again to a second model. Routing saves money on easy requests. Fallback keeps the app up when a model is down.
Model Routing
Sending each request to the model that fits it, cheap or strong.
Model routing picks which model should answer each request before it is sent. A simple rule might send short, easy questions to a small, cheap model. Hard ones, like long code or tricky reasoning, go to a big model. Some routers use a small model of their own to make the choice. The aim is to pay for a big model only when the task needs one.
Model Fallback
Sending a call to a backup model when the first one fails.
Model fallback is a rule that sends a call to a backup model when the first choice fails. A call can fail when the model maker is down, a rate limit is hit, the call takes too long or the reply is broken. Then the app tries the next model on its list, and does not show an error. The backup may be a smaller model, or come from a different firm. The user gets an answer, maybe a bit different, not an error.
In one line
- Model Routing
- Sending each request to the model that fits it, cheap or strong.
- Model Fallback
- Sending a call to a backup model when the first one fails.