AI Gateways: Routing, Fallback, Caching & Metering
One API in front of many model providers: fallback chains that switch only before the first token, circuit breakers, stream watchdogs, exact and semantic caching, token budgets, per-tenant metering, BYOK isolation and TTFT dashboards.
Overview
Why this topic
Once a product calls more than one model, every team writes the same code: retries, timeouts, a second provider when the first is down, a cache, a usage counter. An AI gateway puts that code in one place, between your services and the providers, behind one API (usually the OpenAI chat completions shape).
- Each service holds provider keys — Keys live in the gateway; services hold a gateway token
- A provider outage breaks every feature — The gateway moves traffic to the next model in the chain
- Usage is reconstructed from provider invoices — Every request is metered per tenant, feature and model
- Latency is unknown until users complain — TTFT and error rate per provider on one dashboard
Open-source and hosted examples include LiteLLM Proxy, Portkey, Kong AI Gateway, Envoy AI Gateway and Cloudflare AI Gateway. Many teams build a thin one in-house.
Interview questions
- Design the fallback logic for a streaming chat endpoint that can use three providers. What exactly triggers a fallback?
- Why do you need a stream watchdog when the SDK already has a timeout? How do you implement it?
- How would you add caching to the gateway? When is a semantic cache safe?
- How do you meter usage and enforce budgets per tenant without letting parallel requests overspend?
- Some tenants bring their own provider keys. How do you keep their traffic isolated from the platform keys?
- What would you put on the gateway dashboard, and which alerts would page someone?
Sign up free to read the full topic
More Agentic AI topics
- Async Long-Running Tool Calls
- Autoscaling LLM Inference on Kubernetes
- Browser & Computer-use Agents
- Context Engineering & Prompt Design
- Context Window Management for Persistent Agents
- Designing Tools a Model Calls Well
- Distillation & Small Models
- Embeddings & Vector Search
- Evaluating Agents: Task Suites, Trajectories & Regression Gates
- Evaluating RAG & Catching Hallucinations
- Fine-tuning LLMs: SFT, LoRA/QLoRA & Preference Tuning
- Guardrails & Prompt Injection Defense
Start free on Cariara · Coding problems · System design studio