$ show vllm-project/aibrix
AIBrix is a Kubernetes-native control plane and intelligent gateway for serving production LLM workloads. The repository implements two intertwined planes: a data plane composed of Envoy ext_proc gateway plugins (cmd/plugins, pkg/plugins/gateway) that intercept OpenAI-compatible inference requests and route them to vLLM, SGLang, or TensorRT-LLM replicas using algorithms such as least-load, prefix-cache-aware routing, SLO queueing, token-load tracking, and prefill/decode disaggregation; and a control plane of Kubernetes controllers (cmd/controllers) that reconcile custom resources for autoscaling (PodAutoscaler), LoRA adapters, model warmup/claims, KV caches, RayCluster fleets, and StormService/RoleSet orchestration. Supporting runtime code includes a Python metadata and batch-job service, a distributed KV-cache package, a GPU optimizer, and an OpenAI-compatible frontend.
The LLM identified 5 files worth highlighting for reuse:
Generic provider/handler registry that stores values in an immutable snapshot array and uses atomic copy-on-write to publish updates. Reads never lock; writers lock briefly and swap the snapshot. This pattern is directly reusable in any Go service needing thread-safe plugin registries or dynamic route tables.
Bounded goroutine pool with lifecycle controls (Start/Stop/Submit/TrySubmit/Wait), non-blocking submission, and graceful drain. It is an independent concurrency primitive suitable for any Go batch or job-processing service.
Generic LRU cache with TTL, concurrent eviction, eviction callbacks, and injectable clock. The interface and implementation are isolated from AIBrix and can be reused as a building block for caches in other Go projects.
Byte-preserving JSON editor that mutates request/response bodies without changing their on-wire formatting. It preserves field order, whitespace, and large integer values while injecting routing fields or rewriting prompt payloads. This is non-obvious engineering, useful for gateways/proxies that must hash or stream JSON.
Bounded-concurrency bulkhead and circuit-breaker wrapper around an external HTTP routing decision service. It classifies outcomes, enforces timeouts, and prevents cascading failures. The pattern is reusable for any sidecar or gateway that calls a remote decision endpoint.