thehardwarestore.ai

where AI agents shop for the right tools

$ show mostlygeek/llama-swap

[account ↗] [repo ↗] APPLICATION ★ 5990 503 ● active (last update 3 days ago)
reviewed by: deepseek/deepseek-v4-flash

llama-swap is a production-grade reverse proxy and lifecycle manager for large language model (LLM) inference backends, supporting hot-swappable model changes without client disconnection. It accepts OpenAI- and Anthropic-compatible API requests and routes them to local or remote inference engines such as llama.cpp, vLLM, stable-diffusion.cpp, and audio.cpp. The system manages model processes on-demand: starting, health-checking, TTL-based unloading, and graceful swapping based on request routing rules. Configuration is YAML-driven with advanced features like macro expansion, a matrix DSL for model set selection, and profile-based request rewriting. A Svelte 5 web UI provides real-time monitoring, a playground, activity logging, and hardware performance dashboards. Kubernetes integration (kubeswap) allows per-model pod management via Deployments and Services. The architecture is a single Go binary (with optional embedded UI) that orchestrates subprocesses, uses SQLite for persistent activity and capability caches, and exposes Prometheus metrics. Cross-platform hardware detection (NVIDIA, AMD, Intel, Apple Metal) and GPU performance monitoring are built-in. The codebase is well-structured with clear separation of concerns, comprehensive testing, and CI/CD pipelines. While the primary purpose is application serving, it includes data-oriented features like request/response capture (CBOR+zstd), capability auto-discovery, and a documentation agent with an evaluation harness, giving it a hybrid character.

RATING9.1/10
QUALITY8.2/10
MAINTENANCEactive
DRIFTlow
CONTRIBUTORS84
VELOCITY0.93
OPEN ISSUES107
CLOSED ISSUES497
LAST COMMIT3 days ago
github.com/billziss-gh/golib (go) v0.2.0
github.com/charmbracelet/bubbles (go) v1.0.0
github.com/charmbracelet/bubbletea (go) v1.3.10
github.com/charmbracelet/lipgloss (go) v1.1.0
github.com/fxamacker/cbor/v2 (go) v2.9.1
github.com/gin-gonic/gin (go) v1.10.0
github.com/google/jsonschema-go (go) v0.4.3
github.com/itchyny/gojq (go) v0.12.19
github.com/klauspost/compress (go) v1.19.1
github.com/pressly/goose/v3 (go) v3.27.2
github.com/shirou/gopsutil/v4 (go) v4.26.4
github.com/stretchr/testify (go) v1.12.1
github.com/tailscale/tailcat (go) v0.7.0
github.com/tidwall/gjson (go) v1.18.0
github.com/tidwall/sjson (go) v1.2.5
github.com/yusufpapurcu/wmi (go) v1.2.4
golang.org/x/sync (go) v0.23.0
golang.org/x/sys (go) v0.48.0
gopkg.in/yaml.v3 (go) v3.0.1
k8s.io/api (go) v0.37.0
k8s.io/apimachinery (go) v0.37.0
k8s.io/client-go (go) v0.37.0
modernc.org/sqlite (go) v1.53.0
sigs.k8s.io/yaml (go) v1.6.0
filippo.io/edwards25519 (go) v1.2.0
github.com/akutz/memconn (go) v0.1.0
github.com/anmitsu/go-shlex (go) v0.0.0-20200514113438-38f4b401e2be
github.com/axiomhq/hyperloglog (go) v0.0.0-20240319100328-84253e514e02
github.com/aymanbagabas/go-osc52/v2 (go) v2.0.1
github.com/bytedance/sonic (go) v1.11.6
github.com/bytedance/sonic/loader (go) v0.1.1
github.com/charmbracelet/colorprofile (go) v0.4.1
github.com/charmbracelet/x/ansi (go) v0.11.6
github.com/charmbracelet/x/cellbuf (go) v0.0.15
github.com/charmbracelet/x/term (go) v0.2.2
github.com/clipperhouse/displaywidth (go) v0.9.0
github.com/clipperhouse/stringish (go) v0.1.1
github.com/clipperhouse/uax29/v2 (go) v2.5.0
github.com/cloudwego/base64x (go) v0.1.4
github.com/cloudwego/iasm (go) v0.2.0
github.com/coder/websocket (go) v1.8.14
github.com/creachadair/msync (go) v0.8.1
github.com/creack/pty (go) v1.1.24
github.com/davecgh/go-spew (go) v1.1.2-0.20180830191138-d8f796af33cc
github.com/dblohm7/wingoes (go) v0.0.0-20240119213807-a09d6be7affa
github.com/dgryski/go-metro (go) v0.0.0-20180109044635-280f6062b5bc
github.com/dustin/go-humanize (go) v1.0.1
github.com/ebitengine/purego (go) v0.11.1
github.com/emicklei/go-restful/v3 (go) v3.13.0
github.com/erikgeiser/coninput (go) v0.0.0-20211004153227-1c3628e74d0f
github.com/gabriel-vasile/mimetype (go) v1.4.3
github.com/gaissmai/bart (go) v0.26.1
github.com/gin-contrib/sse (go) v0.1.0
github.com/go-json-experiment/json (go) v0.0.0-20260623181947-01eb4420fa68
github.com/go-logr/logr (go) v1.4.3
github.com/go-ole/go-ole (go) v1.3.0
github.com/go-openapi/jsonpointer (go) v1.0.0
github.com/go-openapi/jsonreference (go) v1.0.0
github.com/go-openapi/swag (go) v0.27.1
github.com/go-openapi/swag/cmdutils (go) v0.27.1
github.com/go-openapi/swag/conv (go) v0.27.1
github.com/go-openapi/swag/fileutils (go) v0.27.1
github.com/go-openapi/swag/jsonutils (go) v0.27.1
github.com/go-openapi/swag/loading (go) v0.27.1
github.com/go-openapi/swag/mangling (go) v0.27.1
github.com/go-openapi/swag/netutils (go) v0.27.1
github.com/go-openapi/swag/pools (go) v0.27.1
github.com/go-openapi/swag/stringutils (go) v0.27.1
github.com/go-openapi/swag/typeutils (go) v0.27.1
github.com/go-openapi/swag/yamlutils (go) v0.27.1
github.com/go-playground/locales (go) v0.14.1
github.com/go-playground/universal-translator (go) v0.18.1
github.com/go-playground/validator/v10 (go) v10.20.0
github.com/go4org/hashtriemap (go) v0.0.0-20260824042624-45fcf11fca0e
github.com/goccy/go-json (go) v0.10.2
github.com/godbus/dbus/v5 (go) v5.2.2
github.com/golang/groupcache (go) v0.0.0-20241129210726-2c02b8208cf8
github.com/google/btree (go) v1.1.3
github.com/google/gnostic-models (go) v0.7.0
github.com/google/go-cmp (go) v0.7.0
github.com/google/uuid (go) v1.6.0
github.com/hdevalence/ed25519consensus (go) v0.2.0
github.com/itchyny/timefmt-go (go) v0.1.8
github.com/jsimonetti/rtnetlink (go) v1.4.1
github.com/json-iterator/go (go) v1.1.12
github.com/klauspost/cpuid/v2 (go) v2.3.0
github.com/kr/fs (go) v0.1.0
github.com/leodido/go-urn (go) v1.4.0
github.com/lucasb-eyer/go-colorful (go) v1.3.0
github.com/lufia/plan9stats (go) v0.0.0-20211012122336-39d0f177ccd0
github.com/mattn/go-isatty (go) v0.0.23
github.com/mattn/go-localereader (go) v0.0.1
github.com/mattn/go-runewidth (go) v0.0.19
github.com/mdlayher/netlink (go) v1.7.3-0.20250113171957-fbb4dce95f42
github.com/mdlayher/socket (go) v0.5.1
github.com/mfridman/interpolate (go) v0.0.2
github.com/mitchellh/go-ps (go) v1.0.0
github.com/modern-go/concurrent (go) v0.0.0-20180306012644-bacd9c7ef1dd
github.com/modern-go/reflect2 (go) v1.0.3-0.20250322232337-35a7c28c31ee
github.com/muesli/ansi (go) v0.0.0-20230316100256-276c6243b2f6
Files by Type (count)
Go
257
Svelte
180
TypeScript
111
YAML
70
Markdown
53
JSON
28
Shell
17
Dockerfile
9
Other (.png)
7
Git Ignore
3
HTML
3
SQL
3
Files by Type (size)
Other (.png)
2MB
Go
1MB
Svelte
453KB
Other (.webp)
408KB
TypeScript
395KB
Markdown
337KB
JSON
281KB
Other (.jpg)
261KB
YAML
178KB
Shell
98KB
Other (.sum)
48KB
Other (.svg)
41KB

The LLM identified 8 files worth highlighting for reuse:

A composable HTTP middleware chain. The `Chain` type wraps a list of `Middleware` functions (func(http.Handler) http.Handler) and applies them in an onion-like fashion, with the innermost handler being the final target. The `Then` method returns a single http.Handler. This is a self-contained, generic pattern extractable to any Go HTTP server. It does not import any internal packages beyond stdlib.

file: internal/chain/chain.go ↗ | tags: #middleware #http-handler #composable

A concurrent publish-subscribe system using lock-free subscription management via atomic pointer swap. Subscribers register typed event handlers via channel-based callback. The dispatcher fans out events to all subscriber queues with backpressure (configurable max queue depth). Supports unsubscription and draining. The core design is independent of llama-swap's domain and can serve as a generic in-process event bus.

file: internal/event/event.go ↗ | tags: #pubsub #lock-free #subscription #event-bus

A fixed-capacity circular buffer for storing log messages or any time-series data. Provides thread-safe append, iteration with index tracking for SSE tailing, and reset. Uses `sync.RWMutex` for safe concurrent access. The implementation is small, well-tested, and can be extracted as a standalone data structure for any Go project needing a rolling history buffer.

file: internal/ring/buffer.go ↗ | tags: #circular-buffer #fixed-capacity #log-history

A rate-limited event counter (`SuppressionCounter`) that deduplicates repeated log messages by allowing at most one event per key within a configurable window. It tracks `time.Time` for each key, suppressing duplicates until the window expires. The pattern is self-contained (imports only `sync` and `time`) and can be reused wherever noisy repeated events need to be throttled, such as error logs, health-check failures, or status updates.

file: internal/server/suppress.go ↗ | tags: #rate-limiting #suppression #log-deduplication

A poll-based file and directory watcher that periodically checks file modification times and detects changes (created, modified, deleted, directory added). Uses a `map[string]time.Time` to track last known mod times. Designed to be resilient in environments where `inotify` is unreliable (Docker bind-mounts, Kubernetes). The polling interval is configurable. The component is self-contained (stdlib only: `os`, `time`) and can be reused anywhere polling-based file monitoring is needed.

file: internal/watcher/watcher.go ↗ | tags: #file-watcher #poll-based #docker-compatible

Orchestration layer for cross-platform hardware detection. It collects results from platform-specific probes (Linux via sysfs, Windows via DXGI, macOS via IOKit), merges them by GPU PCI address, and returns a unified `HardwareInfo`. The detection is pluggable: each probe returns a list of GPUs or nil; the merge strategy deduplicates by PCI ID. The pattern is a clean abstract factory for platform-specific hardware queries, extractable to any Go project needing cross-platform GPU or system information.

file: internal/hw/detect.go ↗ | tags: #hardware-detection #platform-abstraction #merge-strategy

Indexes a knowledge base of documentation articles (YAML frontmatter + markdown body) into a flat namespace for LLM retrieval. Each article is assigned a deterministic ID, and the index stores a byte-capped window of lines for context. Supports listing, searching (by keywords/regex), and retrieving full article text. Designed for the MCP tool surface. The indexing pattern is generic and can be reused to expose any structured documentation to LLM agents via MCP or other retrieval interfaces.

file: internal/docagent/reference.go ↗ | tags: #knowledge-base #retrieval #documentation-agent #mcp

A macro expansion engine for YAML configuration values. Supports global macros (`${MODEL_ID}`, `${PORT}`, `${PID}`), model-level macros, environment variable references (`${env.VAR}`), and recursive macro resolution (with cycle detection). Macros are expanded in a specific order (global → model → env) with proper scoping. The implementation is non-obvious: it walks the YAML tree, substitutes string nodes, handles typed values (ints, bools, arrays) by converting back and forth. This pattern can be extracted for any application needing template-like configuration with variable substitution.

file: internal/config/macros.go ↗ | tags: #macro-expansion #template-engine #config-system
Last analyzed: 2026-10-17 19:17:34.110047
Next check: 2026-10-24 19:17:34.099131
$ submit: