HopBaseThe short answer: high concurrency for LLM APIs is not about how fast a gateway forwards requests. It is three stacked constraints — scarce upstream quotas (hard RPM/TPM caps), upstream errors as a routine condition (429s and 5xxs are daily weather, not incidents), and long-lived requests (streams run for tens of seconds, video jobs for minutes). So availability comes from scheduling: multiple redundant upstreams, failover that treats errors as scheduling signals, and affinity that preserves cache hits under load. A gateway that merely proxies requests inherits every one of those constraints and passes them straight to your application; a gateway that schedules — pooling quota across accounts, retrying classifiable errors against a different upstream, and pinning sessions so prompt caches keep hitting — absorbs them instead. All three mechanisms can be verified before you sign up — a reproducible checklist is at the end.
LLM API concurrency is not classic API concurrency
- Requests live long. A streamed chat holds a connection for tens of seconds; a video generation job runs for minutes. Capacity is consumed by concurrent in-flight connections, not QPS.
- Upstream quota is the scarce resource. Every upstream account has hard RPM/TPM ceilings. No single account survives a production peak; real concurrency comes from the total quota of an account pool.
- Upstream errors are normal. Rate limits, 5xx flapping, and rush-hour congestion exist on every LLM upstream. If the gateway does nothing, every caller absorbs them raw.
Classic gateways compete on throughput. LLM gateways compete on scheduling. Judge them by the scheduler.
Multi-upstream groups: where concurrency actually comes from
HopBase schedules by group: one API key maps to a group backed by multiple upstream accounts and channels, so capacity is the sum of the group quota, not any single account limit. In practice:
- split groups by business line or team, so batch jobs never starve interactive traffic;
- give peak workloads their own group and scale it independently;
- swap models or channels on the gateway side with zero client code changes.
The same mechanism also cuts cost — see group-based routing.
Failover: errors are scheduling signals, not your problem
Most gateways return upstream errors as-is and leave retries to the client — which amplifies upstream instability to every caller. Our approach:
- upstream errors first enter the scheduling verdict: not just 5xx — most classifiable 4xx responses also trigger automatic account and channel failover;
- only after retries are exhausted do we replay the original upstream response — error codes are never swallowed or rewritten into timeouts, so your own backoff logic keeps working;
- billing follows real success: failed requests (non-empty error code) are not billed, which you can verify line by line in the usage export.
Probes and event streams: find problems before users do
- Resident health probes continuously test every channel;
- each upstream account emits an error/recovery event stream — degraded accounts are automatically down-weighted or pulled, and return to the pool after recovery;
- upstream liveness is judged by direct credentialed A/B tests, not vibes.
Cache affinity: concurrency should not cost you prompt cache
Prompt cache hits require requests to land on the same upstream account. Naive load balancing that sprays requests across a pool drives hit rates toward zero — concurrency and cache savings become mutually exclusive by design. The fix is affinity scheduling: a session sticks to one account, the affinity window outlasts the official cache TTL, and scale-out never reshuffles live sessions. Hits pay off twice: discounted cache-read pricing and visibly lower time-to-first-token. Details in our prompt cache guide.
How to verify a gateway under concurrency (reproducible)
- Load-test for distribution: fixed task set, temperature 0, N concurrent runs; read TTFT P95/P99, not the mean — a fat tail means queueing.
- Long-output truncation rate: request long outputs and measure silent mid-stream cutoffs, the worst failure mode there is.
- Error passthrough: deliberately trigger rate limits or invalid parameters and compare error codes with official behavior.
- Cache hit rate: repeat requests with one long system prompt and watch the cached-read share in usage.
- Retest across time windows: peak and off-peak, to catch time-of-day degradation.
Verifying model identity and quality (against swapping and silent nerfs) is a separate method: the fidelity test.
FAQ
How much concurrency does HopBase support?
Capacity is sized per workload — batch pipelines, shared development environments, and multi-business peaks are the design targets. Talk to us before onboarding and we will size a group to your load.
Why not just promise 99.99% SLA?
A number on a landing page cannot be verified by you. We prefer reproducible verification methods and fully disclosed channel attributes (every channel and its rate, published). Contractual service terms are handled with our consultants.
What happens if every upstream fails?
Multi-channel redundancy turns single-point failures into partial degradation. In the extreme case, once retries are exhausted, the gateway replays the original upstream error — no fabricated success, and failures are never billed.
Does scaling out hurt cache hit rates?
No. Affinity pins sessions to accounts; scale-out adds capacity for new sessions without reshuffling existing ones.
For enterprise onboarding and concurrency sizing, contact us; integration details in the docs.