Model Gateways in Production
Tools like LiteLLM, OpenRouter, and Portkey implement the gateway pattern in practice — one interface across many providers.
Prerequisites
Overview
The LLM gateway pattern is provider-neutral in theory; in practice, most teams implement it using an existing tool — LiteLLM, OpenRouter, or Portkey — rather than writing the multi-provider routing and fallback logic from scratch.
Where It Fits
Application
Gateway Tool
One interfaceProvider A
Provider B
Key Points
- Unified interface
- These tools expose one API shape (often OpenAI-compatible) regardless of which underlying provider actually serves the request.
- Automatic fallback
- A gateway tool can retry a failed request against a different provider automatically, improving reliability.
- Cost and usage tracking
- Most gateway tools log every request, making cost attribution and usage monitoring far easier than aggregating logs from multiple providers.
Interview Question
What would you actually lose by writing your own provider-switching logic instead of using a tool like LiteLLM?
Mostly the maintenance burden of keeping up with each provider’s API differences and quirks as they change, plus the built-in fallback, retry, and usage-tracking logic those tools already provide. It’s not that it’s impossible to hand-roll — it’s that a maintained gateway tool has already solved problems most teams would otherwise rediscover.
Explain It in 30 Seconds
Tools like LiteLLM, OpenRouter, and Portkey implement the AI gateway pattern in practice — one interface across multiple providers, with built-in fallback and usage tracking — so most teams adopt one rather than writing that routing logic themselves.
Real-World Stack
Technologies commonly used to implement this in production.