HopBaseThe short answer: LLM API performance comes down to three measurable things — the P95/P99 distribution of time-to-first-token (TTFT), streaming stability (the share of long outputs that finish without a silent mid-stream cutoff), and how prompt-cache hits amplify both speed and cost. Total latency and averages will mislead you: generation time is dominated by output length, and a mean hides the tail your users actually feel. A useful benchmark is boring on purpose — a fixed prompt set, temperature 0, at least 30 runs in one time window, percentiles instead of averages, and the same test repeated against official direct access in the same window. Performance should not be an adjective on a vendor page; it should be a number in your terminal. The method below starts from a single curl command and is fully reproducible.
Why total latency is a bad metric
Total generation time is dominated by output length: 2000 words take ten times longer than 200 regardless of link quality. In streaming, what users actually feel is two things: when the first token arrives (TTFT), and whether tokens keep flowing steadily after it. TTFT is the true link metric — it contains the entire overhead of ingress, gateway scheduling, and upstream queueing.
Read distributions, not means
Averages hide the tail: 90 requests at 1s plus 10 at 8s average to a decent-looking 1.7s, yet every tenth user hits 8 seconds. A fat P95/P99 means queueing or retries — one of the engineering-quality criteria in the fidelity test. Discipline: fixed prompt, temperature 0, at least 30 consecutive runs in one time window, then read percentiles.
Truncation rate: the most ignored metric
A silent mid-stream cutoff on long output is the worst failure mode in streaming: you get half the content and no error code. The test is simple — request long outputs (say 2000 words), repeat N times, and measure the share of streams that finish cleanly with a proper SSE termination. A channel with a high truncation rate makes any TTFT number meaningless.
Cache hits: performance and cost are the same coin
On a prompt-cache hit, input is billed at a discount and TTFT drops visibly — performance and cost are one thing here. Hit rate depends on channel topology: requests must land on the same upstream account, so randomly routed pools converge to zero. How affinity scheduling preserves hit rates under load: our high-concurrency engineering notes; the billing mechanics: the prompt cache guide.
Error codes and retry behavior
An easily missed item: does the gateway pass upstream error codes through untouched? A channel that swallows 429s and turns them into timeouts breaks your backoff logic and collapses under peak load. Test it by deliberately triggering rate limits or invalid parameters and comparing behavior with official docs.
A benchmark that starts with one command
TTFT is measurable with plain curl (with streaming on, time_starttransfer is the first-chunk arrival):
curl -N -s -o /dev/null \
-w "TTFT %{time_starttransfer}s | total %{time_total}s\n" \
https://api.hop-base.com/v1/chat/completions \
-H "Authorization: Bearer $HOPBASE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"","stream":true,"messages":[{"role":"user","content":"Introduce yourself in one sentence"}]}' Loop it 30 times, repeat during peak hours, and you have P95 and jitter. Full discipline: fixed task set, temperature 0, drop the first cold run, retest across time windows, and always compare official direct access against the channel under test in the same window.
How we instrument ourselves
HopBase records model, usage, latency, and status for every request — the same trail that powers billing audits powers our performance monitoring; resident probes continuously test every channel so anomalies are handled before users notice. That is why we publish the method: the numbers you measure and the numbers we watch come from the same link.
FAQ
What counts as a good TTFT?
Absolute values depend on model, region, and ingress. What is comparable is a same-window A/B against official direct access — the gap and the stability of the distribution say more than any absolute number.
Is a gateway slower than going direct?
The fixed cost of one extra hop is milliseconds; the gains from less queueing and cache hits are hundreds of milliseconds to seconds. Run both sides with the method above and let the numbers decide — no need to trust anyone.
How do I keep noise out of the test?
Temperature 0, fixed prompts, N of at least 30, drop cold starts, retest across windows — and never compare across different time windows, or upstream time-of-day swings will masquerade as channel differences.
For enterprise onboarding and comparative benchmarks, contact us; integration details in the docs.