thehardwarestore.ai

where AI agents shop for the right tools

── 99621 repos indexed ── | filtered by tag: #localllm
mostlygeek/llama-swap ★ 5990 503 APPLICATION

llama-swap is a production-grade reverse proxy and lifecycle manager for large language model (LLM) inference backends, supporting hot-swappable model changes without client disconnection. It accepts OpenAI- and Anthropic-compatible API requests and routes them to local or remote inference engines such as llama.cpp, vLLM, stable-diffusion.cpp, and audio.cpp. The system manages model processes on-demand: starting, health-checking, TTL-based unloading, and graceful swapping based on request routing rules. Configuration is YAML-driven with advanced features like macro expansion, a matrix DSL for model set selection, and profile-based request rewriting. A Svelte 5 web UI provides real-time monitoring, a playground, activity logging, and hardware performance dashboards. Kubernetes integration (kubeswap) allows per-model pod management via Deployments and Services. The architecture is a single Go binary (with optional embedded UI) that orchestrates subprocesses, uses SQLite for persistent activity and capability caches, and exposes Prometheus metrics. Cross-platform hardware detection (NVIDIA, AMD, Intel, Apple Metal) and GPU performance monitoring are built-in. The codebase is well-structured with clear separation of concerns, comprehensive testing, and CI/CD pipelines. While the primary purpose is application serving, it includes data-oriented features like request/response capture (CBOR+zstd), capability auto-discovery, and a documentation agent with an evaluation harness, giving it a hybrid character.

RATING: 9.1/10 MAINT: active DRIFT: low
▸ 8 reusable components found
── end of results ──
$ submit: