LLM Routing: When to Put Claude, Gemini, and OpenAI Behind One Gateway

The problem with a single provider

Most AI applications start the same way: pick a model, hard-code the API call, ship it. That works fine until the model you chose gets deprecated, its price doubles, a competitor releases something significantly better, or it goes down at the worst possible moment. Any of these events requires a code change across every place you called the API directly.

The fix is to keep the model choice out of application code. In our own products that means one thin wrapper per app with the model name in one place; once several services share providers, a gateway is the next step. With a gateway, the application code sees one endpoint and the gateway decides which provider gets the request.

How the gateway works

We've self-hosted LiteLLM for internal tools, and it's our default recommendation for a gateway. It presents an OpenAI-compatible API so the application code never changes — you call POST /chat/completions with a model name, and LiteLLM translates that into the right provider API call.

OpenRouter is the managed alternative — one API key, many providers, with fallback handled for you. BrightPath calls OpenRouter directly, with Gemini Flash as the default for Pax, Writing Coach and weekly parent reports; the provider is one environment variable, with direct Claude, OpenAI and MiniMax paths as alternates.

A gateway gives you:

• Request logging. Every call, latency, token count, and cost is recorded. This is how you catch a prompt change that accidentally doubled token usage.
• Rate limit handling. The gateway retries on 429s and switches providers on hard limits. The application sees a result, not an error.
• Cost tracking per project. Tag requests with a project ID and generate weekly cost reports. Without this, costs drift invisibly.
• Zero-downtime model upgrades. When a better model ships, you change one config value instead of hunting through API call sites.

Our model routing rules

Not all tasks need the same model. Route based on three criteria: reasoning requirement, latency tolerance, and cost per call.

Claude Sonnet — reasoning and long context

We default to Claude for tasks that require sustained reasoning, careful instruction following, or long context windows. This includes: writing first drafts from complex briefs, contract and document analysis, and any task where a wrong output has meaningful downstream consequences.

Claude is our most expensive default, but the quality floor is higher than the alternatives on tasks where the prompt is complicated and the output needs to be consistent.

Gemini Flash — high-volume, fast, cheap

For high-throughput tasks where latency matters and cost compounds fast — real-time classification, suggestion generation, short completions — Gemini Flash is usually the right call. It’s significantly cheaper per token than Claude Sonnet and fast enough for interactive use. BrightPath uses it for Pax, its in-lesson tutor: it needs to be fast and cheap at volume.

GPT-4o — when client infrastructure requires it

GPT-4o is the sensible default on client engagements where the client already has Azure OpenAI credits, enterprise agreements, or compliance requirements that tie them to Microsoft infrastructure. The gateway makes this a config change, not a rebuild.

When to introduce routing

You don’t need a gateway on day one. If you have one model doing one task, a direct API call is fine. Introduce a gateway when:

• You have more than two distinct AI workflows with different requirements
• You’re spending more than $500/month on model API calls
• You need request logging for debugging or compliance
• You want to experiment with model switching without code changes

Setting up LiteLLM takes about a day, and we stand it up in our AI infrastructure engagements when a client runs several AI workflows.

One thing people get wrong

Teams often try to pick “the best model” once and use it everywhere. There is no best model — there is the right model for a given task at a given cost point. The goal of a routing layer is to match tasks to models correctly, not to standardise on a single provider out of preference or familiarity.

If you’re running the same frontier model for both a 10,000-token document analysis and a two-sentence category label, you’re probably overpaying on one and correctly spending on the other.