An LLM gateway sits between your app and providers

Every agent project that calls more than one LLM provider needs a gateway: a layer that routes requests, tracks cost, handles failures, and caches responses. Without one, each provider integration is bespoke code that duplicates auth, retry logic, and cost accounting. A gateway consolidates this into a single proxy that speaks one API on the front and many on the back.

The core value is operational. Cost tracking happens at the request level: per user, per project, per model, with budget alerts and spend limits. Routing sends cheap tasks to cheap models and expensive reasoning to frontier models. Fallbacks survive provider outages automatically. Caching eliminates redundant calls to the same prompt. Rate limiting prevents runaway spend from a buggy agent loop. None of these features change what the model returns; they change whether you can afford to run it in production.

LiteLLM is the dominant open-source option (50k+ stars), unifying 100+ providers behind an OpenAI-compatible proxy with all of the above. Portkey adds response caching and unified observability on top. Helicone doubles as both observability (see: tracing and evaluation serve different jobs) and a lightweight gateway via its proxy model. The routing decision (see: model choice trades quality for cost and speed) is the gateway’s most strategic function: the right gateway makes multi-model architecture trivial, and multi-model architecture is what keeps cost under control as usage scales.