LLM API Latency: The Three Numbers That Matter and How to Measure Them (2026)

Latency is the complaint that arrives before any other. A model answers correctly, the cost is acceptable, and the product still feels broken because the first word takes four seconds to appear. Reasoning about LLM API latency as a single number is the reason most attempts to fix it go nowhere.

The three numbers that make up LLM API latency

A single "response time" figure hides three very different quantities.

Time to first token (TTFT) is how long the caller waits before anything comes back. It covers request transit, queueing at the provider, and the prefill pass over your prompt. This is the number users actually feel in a chat interface.

Inter-token latency is the gap between tokens once generation starts, usually reported as its inverse: tokens per second. It determines whether text streams at a comfortable reading pace or stutters.

Total completion time is TTFT plus generation time for every output token. It is the only number that matters for a batch job, and the least useful one for an interactive UI.

Optimising the wrong one is common. A team that shortens total completion time by switching to a smaller model may leave TTFT untouched, because TTFT was dominated by queueing rather than by the model.

What actually drives the numbers

Output length dominates. Generation is sequential — each token waits for the one before it. A 900-token answer takes roughly three times as long to finish as a 300-token answer on the same model. Prompt length behaves differently: prefill is parallel, so doubling a prompt raises TTFT far less than doubling the output raises completion time. If you have one lever, it is the length of what you ask for.

Model size sets the floor. Larger models generate fewer tokens per second. No amount of infrastructure work moves a model below its own generation speed.

Streaming changes the felt experience, not the total. Streaming does not make the response finish sooner. It moves the perceived wait to TTFT, which is usually a fraction of the total. For anything a person reads as it arrives, this is the single highest-leverage change available. The mechanics of consuming the event stream correctly are their own topic — see Claude API streaming.

Distance and network are real but usually smaller. A cross-continent round trip adds tens to low hundreds of milliseconds. That matters when TTFT is 400ms and is noise when TTFT is four seconds.

Provider-side queueing is the invisible variable. Under load, requests wait before prefill begins. This is what turns a well-behaved p50 into an ugly p99, and it is the component you cannot see from inside your own process.

Prompt caching cuts prefill, not generation. Where a provider supports it, a cached prefix removes most of the prefill cost on repeat calls with a long shared preamble. It reduces TTFT and leaves tokens per second unchanged.

How to measure LLM API latency honestly

Most latency numbers people quote are averages taken from a developer laptop against a warm endpoint with an empty prompt. They are not wrong so much as unrelated to production.

Measure percentiles, not averages. The average is dragged around by a small number of very slow calls, which is exactly the population you are trying to control. Report p50, p95 and p99; the gap between p50 and p99 tells you how much queueing you are exposed to.

Record TTFT and total separately. Collapsing them makes it impossible to tell a slow model from a busy one.

Measure with production-shaped inputs. Real prompt lengths, real output lengths, real system preamble. A benchmark that asks for ten tokens tells you almost nothing about a feature that generates six hundred.

Measure from where the calls originate, not from your desk — a serverless region on another continent produces different numbers than your laptop.

Hold concurrency constant and record it. Latency measured at one concurrent request and latency at fifty are different metrics wearing the same name.

What you can actually change

In rough order of effect:

• Ask for less output. Cap max tokens, request structured or summarised answers, and stop asking models to restate the question before answering. This is usually worth more than every other item combined.
• Stream anything a human reads. It converts a total-time problem into a TTFT problem.
• Pick the smallest model that passes your evaluation. Not the smallest available — the smallest that still gets the answer right, measured on your own cases.
• Cache the stable prefix where the provider supports it, especially with long system instructions reused across calls.
• Parallelise independent calls rather than chaining them. Three sequential calls of 1.5s are 4.5s; run concurrently they are about 1.5s.
• Move the caller closer to the provider region, once the items above are exhausted.

When a gateway sits in the path

Routing through a gateway or proxy adds a hop, and the honest expectation is single-digit to low tens of milliseconds of overhead when the gateway is well placed. Against a TTFT measured in hundreds of milliseconds or seconds, that overhead is rarely the problem — but it is worth verifying rather than assuming, because a badly located gateway can add a full cross-region round trip. Compare TTFT with and without the gateway in the path, using the same prompt and the same concurrency. Background on what these components do is in LLM API gateway and Claude API proxy, and on protocol compatibility in OpenAI-compatible API.

Frequently asked questions

What is a good TTFT for an LLM API?

There is no universal figure, and any number quoted without a model, prompt size and region attached is not usable. The practical test is your own p95 against your own interface: if a person is watching a cursor blink, sub-second feels responsive and multi-second feels broken. Set the target from your interaction, then measure whether you hit it.

Does streaming actually reduce latency?

It reduces perceived latency, which is the one users report. Total completion time is unchanged, and in some client implementations marginally worse. For interactive text it is still the highest-value change most teams can make.

Why is my p99 so much worse than my p50?

Almost always queueing — either at the provider under load, or in your own connection pool. A p99 several times the p50 with stable input sizes points at contention rather than at the model. Check whether the spikes cluster in time; if they do, it is load, not content.

Does a longer prompt slow the response as much as a longer answer?

No, and the asymmetry is large. Prefill over the prompt runs in parallel; generation of the answer is strictly sequential. Trimming 500 tokens of output saves considerably more wall-clock time than trimming 500 tokens of prompt.

Is switching to a smaller model always faster?

It raises tokens per second, so generation finishes sooner. It does not help if your latency is dominated by queueing or by network distance, and it costs accuracy. Measure where the time is actually going before trading quality for speed.

The short version

LLM API latency is three numbers, not one: time to first token, tokens per second, and total time. Output length and model choice set the floor; queueing sets the tail; streaming changes what users feel without changing the total. Measure p95 with production-shaped prompts from where your code actually runs, and fix the number that matches your interface rather than the one that is easiest to improve.