Model Orchestration: Routing by Task, Cost and Latency

A provider has a bad afternoon. The status page says degraded performance. Your onboarding assistant, your lead scoring job and your support summariser stop in the same minute, because all three call the same endpoint with the same key and have no idea there is an alternative.

Nobody decided that. It was the default in the first prototype and was never revisited. A single point of failure you chose without a meeting is still a single point of failure.

One provider in the code is a commercial position, not a technical one

Hardcoding a provider fixes three things you would rather keep negotiable: your availability ceiling, your unit price, and your exposure to somebody else's roadmap. Prices move, tiers get renamed, models get deprecated on a few months' notice, and rate limits tighten at the moment your usage grows into them.

A gateway is the boring answer. One internal interface, several providers behind it, routing decided by policy rather than by whichever import statement was written first. It does not make you provider-neutral, but it does make a bad quarter with one vendor a configuration change rather than a project.

The routing key is not "which model is best"

There is no best model, only a fit against five constraints, decided per call rather than per company.

Task class. Classification, extraction, routing and reformatting are pattern work. Judgement, long-context synthesis and anything where a wrong answer is expensive are not.

Quality floor. A score on the eval set for that task, not a feeling. Below the floor a route is invalid regardless of price.

Latency budget. A user waiting on a page has about a second. A nightly enrichment job has an hour, and should spend it on a cheaper tier.

Cost ceiling. Per call and per thousand calls, because the second number is the one on the invoice.

Data residency. Whether this payload may leave a jurisdiction or a tenancy at all. Some of it may not, and that decides the route first.

Run this check in ten minutes. List your live LLM calls and give each a task class, a daily volume and a latency requirement. Most teams find the bulk of their token volume is classification and extraction running on their most expensive tier, because that is what the prototype used.

Cheap models for the volume, expensive models for the judgement

The spread between tiers is roughly an order of magnitude in price and often more in latency, and the quality difference is not uniform across tasks. On constrained work with a clear schema, a small model frequently matches a large one closely enough that the eval set cannot separate them. On open-ended reasoning over a long, messy context the gap is obvious.

So route by what the task needs. Intent classification on inbound forms, field extraction, deduplication hints, routing a ticket: small and fast. Drafting an account plan from twelve sources, reconciling contradictory research, anything a person acts on without reading the working: large. Decide with the eval set per task rather than by reputation.

A silent downgrade is worse than an error for some tasks

Fallback needs two policies, not one. On a transient failure, retry with backoff and jitter against the same provider. On a sustained failure or a rate limit, move to a secondary in a different family, since the incident that takes one provider down often takes its whole region.

What happens next is the question that matters. For background summarisation, degrading quietly to a lower tier is correct: the work completes, slightly worse. For anything with a strict output contract, anything customer-visible, or anything nobody reviews before it acts, a quiet downgrade is wrong. Fail loudly, queue the work, let a human decide. A wrong answer on time costs more than a late one.

Either way, record the model that served every response. Without that field your quality data silently mixes three models and the next eval comparison is meaningless. It belongs beside the prompt version in the log line.

Caching is safe in one place and dangerous in another

Exact-match caching on a key of prompt version, input values, model and parameters is safe, and on repeated classification of the same entities it removes a surprising share of traffic. Provider-side caching of long static prefixes is the other easy win, since most of a system prompt repeats on every call.

Semantic caching, where a similar question returns a previous answer, is where teams get hurt. Two questions that embed closely can need different answers, especially when the answer depends on who is asking or on data that changed this morning. Never share an entry across users whose permissions differ, and tie each entry's lifetime to how fast the underlying data moves.

The bill is the thing that surprises teams

Token cost per call looks trivial in a notebook. Multiplied by a feature that works, it becomes a line item somebody in finance asks about. The usual causes are a retry loop nobody bounded, a prompt that grew by a few thousand tokens of context over six months, and a feature that succeeded.

Put a spend cap per feature at the gateway, not just an account-level ceiling, and decide in advance what happens when it is reached: downgrade, queue, or refuse. That is a product decision, it differs per feature, and it has to be made while calm. An account-level cap alone protects the company by taking down everything at once.

Cost per feature, not cost per month

A monthly invoice is a number finance reads after the fact. The numbers an engineer can act on are per feature and per request: input and output tokens, cost per thousand requests, latency at the median and the ninety-fifth and ninety-ninth percentiles, fallback rate, cache hit rate and error rate by provider.

Percentiles matter more than averages because model latency has a long tail, and the average hides the calls that made a user leave the page. Put those figures beside feature usage in one view: the question is not what you spent, it is which feature is worth what it costs. That is a command centre question, not a billing one.

What the abstraction layer costs you

Two things, both real. You lose provider-specific features or reimplement them: structured output modes, tool-calling schemas, prompt caching semantics, batch endpoints and safety controls all differ, and the common denominator is smaller than any one provider.

And prompts are not portable, whatever the marketing says. The same text against a different family produces different output, often in small ways that pass a glance and fail an eval, so switching means re-tuning and re-testing rather than editing a value. Vendor-neutral means switchable at a known cost, not free.

The gateway is also a dependency of its own. Everything routes through one service you wrote, so keep it thin, keep it stateless, and leave a documented direct path for the day it is the broken thing.

Frequently asked questions

What is an LLM gateway and why use one?

A gateway is a single internal interface in front of several model providers. Applications call it instead of a provider directly, and it decides which model serves each request by policy: task class, quality floor, latency budget, cost ceiling and data residency. It also handles retries, fallback, caching, spend caps and logging, so changing provider, adding a cheaper tier or surviving an outage becomes a configuration change rather than a code migration.

How do you decide which model to use for a task?

Start from the task class rather than the model. Classification, extraction, routing and reformatting are constrained work where small fast models often match large ones closely enough that an eval set cannot tell them apart. Judgement, long-context synthesis and anything acted on without review need a higher tier. Set a quality floor on the eval set for that task, then take the cheapest route that clears it inside the latency budget.

Should an AI system fall back to a weaker model automatically?

It depends on the task, and should be explicit per feature. For background work where a slightly worse result is acceptable, a quiet downgrade keeps the system running. For anything customer-visible, anything with a strict output contract, or anything nobody reviews before it takes effect, a silent downgrade hides a quality change inside a success response, and failing loudly with a queued retry is safer. Record which model served each response either way.

How do you control LLM costs in production?

Measure per feature and per thousand requests rather than per month, since a monthly total cannot be acted on. Enforce spend caps at the gateway per feature, with a decided behaviour at the limit: downgrade, queue or refuse. Route high-volume constrained tasks to cheaper tiers, cache exact repeats and bound retries. The common overruns are unbounded retries, prompts that quietly grew their context, and features that became popular.