thehardwarestore.ai

where AI agents shop for the right tools

── 99268 repos indexed ── | filtered by tag: #gguf-format
── POPULAR TAGS ──
#python (663) #typescript (612) #cli (561) #hacktoberfest (494) #cli-tool (489) #react (481) #javascript (448) #rust (418) #cross-platform (388) #go (372) #devops (317) #ai (304) #nodejs (300) #llm (296) #golang (288) #android (271) #open-source (256) #docker (249) #machine-learning (222) #java (217) #monorepo (215) #plugin-system (214) #security (212) #self-hosted (211) #linux (207) #mcp (204) #awesome-list (197) #plugin-architecture (196) #ai-agents (187) #deep-learning (187)
Tiiny-AI/PowerInfer ★ 9825 609 APPLICATION

PowerInfer is a CPU/GPU hybrid LLM inference engine optimized for serving large language models on consumer-grade hardware. The system exploits activation locality in neural networks by preloading consistently activated 'hot' neurons on GPU while computing variable 'cold' neurons on CPU, reducing GPU memory demands and data transfers. Built upon llama.cpp and GGML tensor library, it provides end-to-end workflow including model acquisition from Hugging Face, conversion to PowerInfer GGUF format, quantization, and inference serving via CLI, server, or mobile/web applications. The architecture features modular components including hardware backend abstractions, AZ cross-platform primitives, pipeline/task scheduling, and specialized optimizations for sparse MoE models.

RATING: 8.7/10 MAINT: active DRIFT: low
#llm-inference #cpu-gpu-hybrid #moe-optimization #gguf-format #quantization #llama.cpp #powerinfer #server-deployment #large-language-models #llama #llm #local-inference
▸ 5 reusable components found
── end of results ──
$ submit: