$ show vllm-project/semantic-router
This repository implements a production-grade semantic router for LLM inference, functioning as an Envoy ext_proc filter that intercepts chat completion requests, classifies them via multiple signals (intent, PII, jailbreak, domain, embeddings, etc.), and routes to the most appropriate model or model composition. The system supports Mixture-of-Models (MoM) with configurable algorithms (fusion, cascading, workflows) and provides advanced features like semantic caching, context compression, rate limiting, and observability. It includes a full-stack dashboard (React + Go backend) for configuration management, evaluation/experimentation (campaigns, A/B tests, capacity SLOs), and monitoring. A Python CLI and Kubernetes operator are available for deployment. The codebase demonstrates high engineering quality with extensive validation, testing, and a modular architecture, though its polyglot nature (Go, Rust, Python, TypeScript) and numerous optional backends introduce maintenance complexity. While research elements (training scripts, benchmarks) exist, the primary character is a deployable application.
The LLM identified 5 files worth highlighting for reuse:
A domain-specific language compiler that parses routing policies into structured RouterConfig. Includes decompilation back to DSL, scoped symbol tables, and round-trip validation tests. Self-contained and could be extracted for other config-driven systems.
A hybrid semantic cache with exact L1 and semantic L2 tiers, supporting multiple backends (in-memory, Milvus, Qdrant, Valkey, Llama Stack). Includes polarity guard, TTL, and consistency levels. Well-abstracted with clean interfaces and could be reused in other LLM proxy systems.
A set of functions for sanitizing logs before storage or display, covering credential redaction (passwords, tokens, API keys), URL cleanup, and private key masking. Easily extractable for any application that logs sensitive data.
A continuous batching scheduler for embedding inference, managing request queues, dynamic batching, and resource allocation. Demonstrates sound concurrency patterns and is decoupled from specific model implementations, making it reusable for any batched ML inference workload.
Implements paired live evaluation campaigns using controlled A/B comparisons. Handles identity reservation, outcome reduction, evidence attestation, and readiness gates. While tied to the dashboard backend's domain, the protocol and state machine pattern are transferable to other evaluation systems needing robust comparison protocols.