LLM Inference Gateway + AI SRE Control Plane

A multi-provider LLM gateway with a reliability control plane in front of it.

This system is in progress. The case study reflects the current design and intent; metrics are labeled as targets or planned until measured.

Problem

Teams calling LLMs directly couple product code to a single provider, with no shared caching, quota enforcement, failover, or cost visibility. When a provider degrades, everything downstream degrades with it.

Why this matters

Every lab and applied-AI company serves models behind a gateway. Routing, caching, quotas, failover, and the SRE control plane to operate them are exactly the reliability and cost problems that inference platform teams at frontier and big-tech AI orgs own at scale.

Constraints

Provider-agnostic: no lock-in to a single vendor SDK.
Low added latency budget on the hot path.
Cost-controlled by default: quotas and mock mode over real spend.
Portable across k3s (dev) and managed Kubernetes (prod).

Architecture

Clients

Product apps
Console UINext.js

Gateway

Router / failoverGo
Rate limit / quotaRust hot path
Cacheresponse + semantic

Providers

Hosted LLM APIs
Self-hosted models
Mock providerdefault in dev

Control plane

Metrics + traces
Cost accounting
Incident / runbooks

A stateless Go gateway fronts provider adapters and a cache tier; a Rust hot-path component handles token accounting and limits; a Next.js console reads health, cost, and traces from the control plane.

Data flow

Request enters the gateway, checks the cache, passes rate/quota accounting, routes to a provider adapter (with failover), and returns while emitting metrics, cost, and traces.

Control plane vs data plane

Control: Health, cost accounting, quotas, routing policy, and incident/runbook views.

Data: The per-request hot path: cache lookup, token accounting, provider routing, and response streaming.

Core capabilities

Single typed API surface in front of multiple model providers.
Response and semantic caching to cut cost and tail latency.
Per-tenant rate limits, quotas, and budget enforcement server-side.
Golden-signal dashboards, cost accounting, and a safe degraded mode.

Staff-level tradeoffs

Gateway in Go, hot-path limits in Rust.
Go keeps the routing/adapter layer productive; the Rust component keeps per-request accounting predictable under load without GC pauses.
Mock provider is the default in dev.
Demos and tests stay free and deterministic; real providers require explicit keys and quotas.
Cloudflare Tunnel over a cloud load balancer for dev.
Avoids fixed load-balancer cost and keeps the app cloud-neutral while still exposing a public HTTPS hostname.

Tech stack

Frontend

Next.js
React
TypeScript

Backend

Systems

Rust

Infrastructure

Kubernetes
Terraform
AWS
OCI
Cloudflare

Observability

OpenTelemetry
Prometheus
structured logs

Metrics

Added p95 gateway overhead: < 15 ms
Cache hit rate (cacheable traffic): ≥ 40%
Availability SLO: 99.9%
Provider failover: Automatic on 5xx / timeout
Cost per 1k tokens: Tracked per tenant/provider

Metrics are labeled measured, target, or planned. Nothing here is an achieved result unless it is marked measured.

Failure modes

Provider outage or timeout
Automatic failover to a healthy provider; degraded mode surfaced as an observable state.
Cache stampede on a hot key
Request coalescing so only one upstream call fills the cache.
Tenant quota exhausted
Fast, typed rejection with a clear error, so providers never see cascading load.

Lessons & decisions

Made the mock provider the default so demos and tests never depend on paid APIs or the network. That decision shaped the adapter interface first.
Prototyped rate limiting in Go, then moved the hot-path accounting to Rust to keep per-request overhead predictable under load.

What's next

Wire semantic cache and measure hit rate on representative traffic.
Publish a load-test report (throughput, p50/p95/p99, cost).
Add prompt-injection and abuse test suites to the gateway.