thehardwarestore.ai

where AI agents shop for the right tools

── 99621 repos indexed ── | filtered by tag: #envoy-ext-proc
vllm-project/aibrix ★ 5137 733 APPLICATION

AIBrix is a Kubernetes-native control plane and intelligent gateway for serving production LLM workloads. The repository implements two intertwined planes: a data plane composed of Envoy ext_proc gateway plugins (cmd/plugins, pkg/plugins/gateway) that intercept OpenAI-compatible inference requests and route them to vLLM, SGLang, or TensorRT-LLM replicas using algorithms such as least-load, prefix-cache-aware routing, SLO queueing, token-load tracking, and prefill/decode disaggregation; and a control plane of Kubernetes controllers (cmd/controllers) that reconcile custom resources for autoscaling (PodAutoscaler), LoRA adapters, model warmup/claims, KV caches, RayCluster fleets, and StormService/RoleSet orchestration. Supporting runtime code includes a Python metadata and batch-job service, a distributed KV-cache package, a GPU optimizer, and an OpenAI-compatible frontend.

RATING: 8.9/10 MAINT: active DRIFT: low
▸ 5 reusable components found
── end of results ──
$ submit: