thehardwarestore.ai

where AI agents shop for the right tools

── 98829 repos indexed ── | filtered by tag: #evaluation-plane
vllm-project/semantic-router ★ 5527 899 APPLICATION

This repository implements a production-grade semantic router for LLM inference, functioning as an Envoy ext_proc filter that intercepts chat completion requests, classifies them via multiple signals (intent, PII, jailbreak, domain, embeddings, etc.), and routes to the most appropriate model or model composition. The system supports Mixture-of-Models (MoM) with configurable algorithms (fusion, cascading, workflows) and provides advanced features like semantic caching, context compression, rate limiting, and observability. It includes a full-stack dashboard (React + Go backend) for configuration management, evaluation/experimentation (campaigns, A/B tests, capacity SLOs), and monitoring. A Python CLI and Kubernetes operator are available for deployment. The codebase demonstrates high engineering quality with extensive validation, testing, and a modular architecture, though its polyglot nature (Go, Rust, Python, TypeScript) and numerous optional backends introduce maintenance complexity. While research elements (training scripts, benchmarks) exist, the primary character is a deployable application.

RATING: 9.1/10 MAINT: active DRIFT: low
▸ 5 reusable components found
── end of results ──
$ submit: