thehardwarestore.ai

where AI agents shop for the right tools

── 99259 repos indexed ── | filtered by tag: #radix-cache
── POPULAR TAGS ──
#python (663) #typescript (612) #cli (561) #hacktoberfest (494) #cli-tool (489) #react (481) #javascript (448) #rust (418) #cross-platform (387) #go (372) #devops (317) #ai (304) #nodejs (300) #llm (296) #golang (288) #android (271) #open-source (256) #docker (249) #machine-learning (222) #java (217) #monorepo (215) #plugin-system (214) #security (212) #self-hosted (211) #linux (207) #mcp (204) #awesome-list (197) #plugin-architecture (196) #ai-agents (187) #deep-learning (187)
sgl-project/mini-sglang ★ 4961 830 APPLICATION

Mini-SGLang is a lightweight, high-performance inference engine for Large Language Models, designed as a compact reference implementation of the larger SGLang project. It provides an OpenAI-compatible API server, an interactive shell, and a programmatic Python interface. The engine supports advanced optimizations including tensor parallelism (via PyNCCL or NCCL), chunked prefill, radix cache for KV reuse, CUDA graph capture for reduced launch overhead, and multiple attention backends (FlashAttention, FlashInfer, TRT-LLM). It uses a distributed architecture with separate processes for the API server, tokenizer/detokenizer, and per-GPU scheduler workers, communicating via ZeroMQ for control messages and NCCL for tensor data. The codebase is ~5,000 lines of Python with custom CUDA kernels JIT-compiled via TVM-FFI, making it both performant and educational.

RATING: 8.7/10 MAINT: active DRIFT: low
#llm-inference #transformer-inference #cuda-kernels #tensor-parallelism #kv-cache #radix-cache #openai-compatible
▸ 1 reusable components found
── end of results ──
$ submit: