Open-Source LLM Gateway: 4 Projects Compared and How to Choose (2026)

What "open source" actually changes about an LLM gateway

An LLM gateway sits between your application and one or more model providers. It normalizes the request format, routes each call to a provider, and handles the operational work around that call — keys, retries, fallbacks, rate limits, caching, logging, and spend tracking.

Choosing an open-source gateway changes three things, and it is worth being precise about which ones actually matter to you:

Where the traffic terminates. A self-hosted gateway means your prompts and completions pass through infrastructure you run. For teams with data-residency obligations or an internal rule against third-party proxies, this is usually the whole reason for the decision.
What you can change. Routing logic, custom auth, per-team budgets, a bespoke redaction step — all of it is editable rather than filed as a feature request.
What you now operate. This is the part that gets underweighted. A gateway is on the critical path of every model call. Self-hosting means you own its uptime, its upgrade cycle, and its failure modes at 3 a.m.

If you have not yet settled the question of whether you need a gateway at all, start with what an LLM API gateway does and when you need one — the rest of this article assumes you have already answered yes.

The four projects worth a shortlist

The open-source field has consolidated. Four options cover the large majority of real deployments.

LiteLLM

The most widely adopted open-source LLM proxy, and the fastest route from nothing to a working gateway. Its defining strength is provider breadth — well over a hundred providers exposed behind a single OpenAI-compatible surface, which means your client code keeps speaking one protocol regardless of what sits behind it.

The trade-off is its runtime. LiteLLM is Python, and under heavy concurrent load the proxy layer itself becomes a measurable part of your latency budget. For most teams at moderate volume this never surfaces. For high-throughput serving paths it is the first thing to load-test.

Fits when: provider breadth is the priority, or you want a working proxy this afternoon.

Portkey Gateway

MIT-licensed, with fallbacks, retries, and guardrails in the open-source core rather than fenced off behind a paid tier. It began as a hosted product and open-sourced the gateway component, which shows in the design — the routing and reliability primitives are more opinionated and more complete than a proxy that grew organically.

It also ships semantic caching, which matches on prompt similarity rather than exact string equality. Whether that saves you meaningful money depends entirely on how repetitive your traffic is; treat vendor-published savings percentages as marketing, and measure your own cache hit rate before counting on it.

Fits when: you want reliability primitives (fallback chains, retries, guardrails) as first-class configuration.

Envoy AI Gateway

Built on Envoy, Apache 2.0 licensed. This is the pick for teams already standardized on Envoy and Kubernetes — it inherits the proxy layer you already run, already monitor, and already know how to debug.

The reasoning here is mostly organizational rather than technical. If your platform team operates Envoy, adding an AI-specific filter is a smaller change than introducing a new Python service with its own deployment story and its own on-call surface.

Fits when: Kubernetes and Envoy are already your standard, and the gateway should not be an exception to it.

Kong and Apache APISIX AI plugins

Both mature API gateways now ship AI-proxy plugin layers. The logic is the same as the Envoy case: if you already run an API gateway for the rest of your traffic, routing model calls through it keeps auth, rate limiting, and observability in one place instead of two.

Fits when: you already operate one of these and want one gateway, not two.

Five decisions that pick the project for you

Feature matrices are close to useless here — every project lists the same nouns. These five questions actually separate them.

1. What is your latency budget, and have you tested against it? A gateway adds a hop. The question is whether that hop is 3 ms or 40 ms at your concurrency. Load-test with your real payload sizes and real concurrency before committing; a runtime that is fine at 10 requests per second may not be fine at 500.

2. How many providers do you actually need? Teams routinely shortlist on "supports 100+ providers" and then ship with two. If you need two, breadth is not a differentiator and you should weight runtime and operability instead.

3. Where does it deploy? A Python service, a Kubernetes filter, and an edge-deployable worker are three different operational commitments. The right answer is almost always "the same place your other infrastructure already lives."

4. What does "observability" mean in this project? The word covers everything from a request log to per-team token accounting with cost attribution. If you need chargeback by team, verify that specific capability exists rather than trusting the bullet point. Cost attribution is the feature most often thinner than advertised.

5. Who is on call for it? The gateway is on the critical path of every model call. If the answer to "who gets paged when it fails" is unclear, that is an argument for either the option your platform team already operates, or a managed endpoint.

What self-hosting actually costs

The software is free. The commitment is not:

Availability. You are now the reason model calls fail, or don't. Redundancy and failover are your build.
Upgrades. Provider APIs change. Someone tracks those changes and ships the update.
Key custody. Provider keys now live in your infrastructure. That is often the point — but it is also a secret-management responsibility you have absorbed.
Debugging. When latency spikes, you own the question of whether it was the provider or your hop.

None of this argues against self-hosting. It argues for counting it honestly against the alternative, rather than treating "open source" as "free."

When a hosted endpoint is the better answer

Self-hosting a gateway earns its keep when you need control — over data paths, routing logic, or deployment location. It earns much less when what you actually needed was a stable OpenAI-compatible endpoint that already speaks the protocol your tools expect.

If your requirement is closer to "point Claude Code or an OpenAI SDK at a working endpoint and get on with the work," a hosted OpenAI-compatible API removes the operational surface entirely. ROIBest AI provides an OpenAI-protocol endpoint that works with existing clients through a base-URL change, with no gateway to deploy or keep running. Whichever way you go, understanding what actually drives your bill is worth doing before you optimize the wrong layer.

The honest framing: a gateway is infrastructure. Adopt it when you have a problem it solves, not because the category exists.

FAQ

Is an open-source LLM gateway free? The license is. The operating cost is not — compute, on-call coverage, and upgrade maintenance are real line items. Compare total cost against a hosted endpoint rather than comparing license fees.

Can I switch gateways later? Usually, and this is a genuine argument for the category. If every project you shortlist exposes an OpenAI-compatible surface, your application code is insulated from the choice, and swapping is a configuration change rather than a rewrite.

Do I need a gateway if I only use one provider? Often not. With a single provider and no routing, fallback, or per-team accounting requirements, a gateway adds a hop and an operational surface without a corresponding benefit. Revisit when you add a second provider or need spend attribution.

Which one should I start with if I am just evaluating? LiteLLM, for the practical reason that it is the fastest to stand up — which makes it a cheap way to discover what you actually need. Treat that first deployment as a probe, not a commitment.

Does self-hosting mean my prompts stay private? It means they do not transit a third-party proxy. They still go to the model provider, under that provider's data terms. Self-hosting the gateway narrows the set of parties involved; it does not remove the provider from it.