vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
LocalAI is a modular AI inference server providing OpenAI-compatible APIs for running diverse models (LLMs, vision, audio, video) locally. It features a polyglot architecture with Go core services, pluggable backends (C++/Go/Python), and a React-based web UI. The system supports distributed deployment via NATS/PostgreSQL, model gallery management, and multimodal processing. Key capabilities include model loading from Hugging Face, quantization, PII detection, and real-time transcription. The codebase demonstrates strong engineering practices with dependency injection, middleware chains, and clean separation of concerns.
Unsloth is a full-stack, local-first AI platform for fine-tuning and inference of large language models (LLMs), diffusion models, and audio models. It consists of two primary components: (1) the core `unsloth` Python library, which provides highly optimized LoRA/QLoRA/RL training with custom Triton kernels, and (2) the Unsloth Studio application, a Tauri desktop app with a React frontend (TanStack Router, Zustand) and a FastAPI backend (SQLite, structlog). The platform enables users to discover models from the Hugging Face Hub, download and manage them locally, run inference with chat, image/video generation, and RAG, fine-tune with a visual recipe editor, and export in multiple formats (GGUF, NVFP4). It also integrates external LLM providers (OpenAI, Anthropic, Ollama) and supports agentic tool loops via MCP and Codex transports. The codebase is extensively tested with both unit tests and source-level contract tests, and includes cross-platform desktop installers for Windows, macOS, and Linux.
Repomix is a multi-interface tool that packs code repositories into a single AI-optimized file for consumption by large language models (LLMs) like ChatGPT, Claude, and Gemini. It operates as a CLI tool, a library, an MCP (Model Context Protocol) server, a GitHub Action, a browser extension, and a web service. The core pipeline discovers files respecting ignore patterns, reads them with binary and encoding detection, optionally compresses code via tree-sitter (removing implementation details while preserving signatures), scans for secrets with Secretlint, and generates output in XML, Markdown, JSON, or plain text formats. It also provides token counting with configurable encodings and disk-backed caching, git integration (diffs, logs, change-frequency sorting), and skill generation for Claude Agent. Architecturally, the project is modular with a clear separation between the core library (src/core/), CLI interface (src/cli/), MCP server (src/mcp/), website (website/), and browser extension (browser/). The core uses dependency injection for testability, worker threads (tinypool) for parallel file processing and metrics, and a strategy pattern for language-specific tree-sitter parsing. Configuration is loaded from JSON/JS/TS files and merged with CLI options. The codebase demonstrates strong engineering practices including comprehensive test coverage, security hardening (SSRF protection, git command sanitization, sandboxed MCP tools), and performance optimizations (token count cache, bounded concurrency, fast binary detection). The project is actively maintained with frequent releases, a large community (28k+ stars), and extensive documentation in 15 languages. While the core is well-architected, the multiple distribution channels and large dependency surface introduce maintenance overhead. Some duplication exists between the website and CLI validation logic, and the tree-sitter WASM management is fragile. Overall, it is a high-quality, production-ready tool with several reusable components.
Ollama is a local LLM serving platform written in Go. It provides a REST API (OpenAI-compatible), CLI, TUI chat, and desktop application for running large language models locally. The system supports multiple inference backends: llama.cpp (CPU/CUDA/ROCm/Vulkan) and MLX (Apple Silicon). It includes a model conversion pipeline from HuggingFace safetensors to GGUF format, GPU hardware discovery, and an agent framework with tool execution (bash, file operations, web search). The architecture is a monolith with subprocess management for inference backends. Key entry points include the CLI (main.go), HTTP server (server/routes.go), runner binary (cmd/runner/main.go), and desktop UI (app/ui/app/). The codebase is actively maintained with extensive test coverage and support for dozens of model architectures. While primarily an application, it has significant data/research components in model conversion, tokenization, and speculative decoding research.
Aider is a mature, well-architected CLI application that enables AI pair programming in the terminal. It integrates with 100+ LLMs via LiteLLM to help developers edit code in local git repositories. The system is built around a sophisticated coder abstraction that supports multiple edit formats (whole file, diff, search/replace blocks, unified diffs) to accommodate different LLM capabilities. The architecture is plugin-based with a clear separation between the main CLI loop, coder implementations, model management, and repository mapping. Aider automatically commits changes with sensible commit messages, provides linting/testing integration, and supports advanced features like prompt caching, voice input, and a browser-based GUI. The codebase demonstrates strong engineering practices with comprehensive testing, CI/CD pipelines, and careful dependency management. The repo map feature using tree-sitter for code analysis is particularly notable, providing intelligent context about codebases to LLMs. The project maintains high quality through extensive test coverage, pre-commit hooks, and automated benchmarking against coding tasks.
LLaMA-Factory is a production-grade, unified framework for fine-tuning, evaluating, and serving 100+ large language and multimodal models. It provides a CLI (`llamafactory-cli`), a Gradio web UI (LLaMA Board), and an OpenAI-compatible API server. The architecture is monolithic but highly modular, built on Hugging Face Transformers/TRL/PEFT with optional integration of vLLM, SGLang, DeepSpeed, FSDP, Megatron Bridge, and hardware-specific backends (CUDA, ROCm, NPU, XPU). Training pipelines support full-parameter, LoRA, QLoRA, freeze-tuning, and alignment methods (DPO, PPO, KTO, etc.) with extensive data processing (multimodal, packing, streaming, template-based formatting). The codebase features a plugin system (v1) for kernels, quantization, distributed backends, and batching strategies, though it maintains a parallel legacy (v0) path. Reusable components include dataset processors, chat template hierarchy, multimodal plugin architecture, custom trainer callbacks, and batching strategy plugins. While the framework is actively maintained and well-tested, it suffers from some technical drift due to a dual-codebase structure, hardcoded model registries, version pinning, and fragile monkey-patching in distributed integrations.
Lemonade is a local AI inference server and desktop application that provides OpenAI/Anthropic/Ollama-compatible APIs with support for multiple backends (llama.cpp, vLLM, whisper.cpp, Kokoro, etc.). The C++ server core handles model lifecycle, routing policies, multi-modal inference, and telemetry, while the React/TypeScript frontend (Tauri desktop app and web build) offers a full GUI for chat, image/audio generation, transcription, and model management. The repository also includes a CLI client, tray application, extensive integration tests, and automated documentation generation. Architecture follows a clean backend-descriptor registry pattern with generated boilerplate, platform abstraction for process/archive/path operations, and a sophisticated routing policy engine for model selection.
llama-swap is a production-grade reverse proxy and model router for local large language model (LLM) inference servers. It manages the lifecycle of upstream model processes (e.g., llama.cpp, vLLM), routes incoming OpenAI-compatible HTTP requests to the appropriate model based on configurable groups, matrix constraints, and peer definitions, and supports hot-swapping models without restarting the proxy. The system provides a rich web UI (Svelte 5) for monitoring activity, managing models, and running interactive playground tools including a documentation agent that answers questions via MCP tools. It also includes auxiliary utilities like a Wake-on-LAN proxy and a vLLM sleep mode wrapper. The architecture is a single Go binary with embedded static assets, internally modular with distinct packages for configuration (YAML with macros and deep merge), process management (state machine with health checks and TTL), routing (group and matrix DSL), hardware detection, performance monitoring, and activity storage (SQLite). External dependencies are limited to upstream model servers and optional remote peers. The codebase is actively maintained, well-tested, and demonstrates advanced engineering patterns such as a generic event pub/sub system, a ring buffer, composable HTTP middleware chains, a configurable matrix-based model selection DSL, and a poll-based directory watcher suitable for containerized environments.
PocketRisu is a self-hosted AI roleplay chat platform, forked from RisuAI. It runs as a Node.js server with a Svelte 5 frontend, accessed via browser. The system supports multiple AI providers (OpenAI, Anthropic, Google Gemini, etc.) through a pluggable adapter pattern, and includes character management, lorebooks, plugins, MCP (Model Context Protocol) tools, translation, TTS, image generation, and a comprehensive data-driven settings system. Data is stored in SQLite with content-defined chunking for large values, and the architecture is a monolith with a client-server split: the server handles persistence, backup, asset manifests, and model job processing; the client handles UI, state management, and direct AI API calls (with optional server-side proxy). Key entry points include src/main.ts (client bootstrap), server/node/server.cjs (server), src/ts/storage/database.svelte.ts (central store), and src/ts/process/request/ (AI request handling). External dependencies include Node.js, SQLite, various AI provider APIs, MCP servers, and browser APIs (File System Access, IndexedDB). The codebase is well-tested with unit and integration tests, and features a sophisticated plugin system with sandboxed execution, a model preset registry with remote sync, and a chat delta sync mechanism for efficient data transfer.
SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
This repository is a high-performance React Native library (llama.rn) that enables on-device Large Language Model (LLM) inference on iOS and Android devices. It wraps the C++ llama.cpp core using JavaScript Interface (JSI) and TurboModules, allowing mobile applications to run GGUF-formatted models locally without cloud dependencies. The architecture consists of a TypeScript/JavaScript layer for the React Native interface, a C++ core for inference and model loading, and native modules for Android (Java/Kotlin) and iOS (Objective-C++). Key capabilities include hardware acceleration via Apple Metal and Qualcomm Hexagon DSP, multimodal support (vision/audio), parallel decoding, and structured output generation. The project includes a comprehensive example app demonstrating chat interfaces, model management, and benchmarking tools.
H2O LLM Studio is a web-based application for fine-tuning large language models (LLMs) without requiring coding experience. It provides a GUI built on H2O Wave and CLI tools for dataset import, experiment configuration, model training, evaluation, and export to Hugging Face Hub. The system supports multiple problem types including causal language modeling, classification, regression, sequence-to-sequence, and Direct Preference Optimization (DPO). The architecture centers around a Wave server handling both UI and backend logic, with distributed training capabilities via DeepSpeed. Key components include a configuration system using dataclasses, utilities for conversation chain handling, model loading abstractions, and factory patterns for training components.
PowerInfer is a CPU/GPU hybrid LLM inference engine optimized for serving large language models on consumer-grade hardware. The system exploits activation locality in neural networks by preloading consistently activated 'hot' neurons on GPU while computing variable 'cold' neurons on CPU, reducing GPU memory demands and data transfers. Built upon llama.cpp and GGML tensor library, it provides end-to-end workflow including model acquisition from Hugging Face, conversion to PowerInfer GGUF format, quantization, and inference serving via CLI, server, or mobile/web applications. The architecture features modular components including hardware backend abstractions, AZ cross-platform primitives, pipeline/task scheduling, and specialized optimizations for sparse MoE models.
TinyLLaVA Factory is a modular Python framework for building, training, evaluating, and serving small-scale Large Multimodal Models (LMMs). It provides a pluggable architecture with separate components for vision towers (CLIP, SigLIP, DINOv2), language models (Phi, TinyLlama, Gemma, Qwen2, etc.), and connectors (MLP, QFormer, Resampler), all accessible via factory registries. The training pipeline supports two-stage pretraining (alignment) and finetuning with full, LoRA, or QLoRA recipes using DeepSpeed. It includes a comprehensive evaluation suite for standard benchmarks (VQA, GQA, SQA, POPE, MMMU, etc.) and a Gradio-based web demo. The codebase is well-organized and follows clean architectural patterns, though many modules are tightly coupled to the internal framework. It is actively maintained, with recent additions like a visualization tool and support for new models.
AstrBot is a production-grade, multi-platform LLM chatbot framework and development platform. It connects to 10+ instant messaging platforms (QQ, Telegram, Discord, Slack, WeChat, Lark, etc.) via adapter pattern, integrates with 20+ LLM providers (OpenAI, Anthropic, Gemini, DashScope, etc.), and provides a plugin system (Stars) with command/regex/filter decorators. The architecture is a modular monolith: a FastAPI backend serves a Vue 3 dashboard for management, while a pipeline of stages processes incoming messages through waking checks, rate limiting, content safety, LLM interaction, and response delivery. It includes a knowledge base with RAG (FAISS vector store, chunking, hybrid retrieval), agent runners (Dify, Coze, DeerFlow), sandboxed code execution (computer use), TTS/STT, cron jobs, and a CLI. The codebase is well-structured with extensive test coverage, but its broad scope introduces some duplication and deprecated modules.
Fish Speech is a production-grade multilingual text-to-speech system with a dual-autoregressive transformer architecture (4B parameter slow AR for temporal semantics, 400M fast AR for residual codebooks). It supports real-time voice cloning from short reference audio (10-30s), multi-speaker multi-turn generation, and fine-grained prosody/emotion control via natural language tags. The system is deployed through multiple interfaces: a Gradio web UI, a React-based modern web UI, a production ASGI API server (Kui) with streaming support, and command-line tools for inference and batch processing. Its architecture is modular: a TTSInferenceEngine orchestrates reference loading, LLM inference, and VQ decoding; an RVQ audio codec (10 codebooks, ~21 Hz frame rate) compresses audio; and the dual-AR model is structurally isomorphic to standard LLMs, enabling reuse of inference optimizations like paged KV cache and RadixAttention. The repository also includes training infrastructure (Lightning + Hydra) with LoRA fine-tuning, protobuf-based dataset pipelines, and a comprehensive i18n framework. While primarily a deployable application, it retains strong research character with published papers, benchmark results, and custom model implementations.
ProxyAI (formerly CodeGPT) is a JetBrains IDE plugin that provides AI-powered code assistance including chat, inline editing, code completions, agent-based task execution, commit message generation, and next-edit suggestions. The plugin integrates with multiple LLM providers (OpenAI, Anthropic, Google, Mistral, Ollama, llama.cpp, custom OpenAI-compatible, and its own ProxyAI cloud) and supports local inference via a bundled llama.cpp submodule. It implements an agent framework with a tool system (read, write, edit, bash, web search, diagnostics, debugger control), MCP (Model Context Protocol) client for external tool servers, and ACP (Agent Client Protocol) for external agent integration. The architecture follows IntelliJ Platform conventions: services, actions, tool windows, settings via PersistentStateComponent, and editor listeners. The codebase is a mix of Kotlin (dominant) and Java, with Gradle build system and extensive use of coroutines for async operations.
Eidolon is an open-source framework for building, deploying, and managing multi-agent AI applications. It provides a Python SDK with abstractions for LLM integration (OpenAI, Anthropic, Mistral, Ollama, Azure), retrieval-augmented generation (RAG) pipelines, memory systems (vector stores, symbolic memory, file storage), and document processing (loaders, parsers, splitters). Agents are configured declaratively via YAML and can communicate with each other, use tools, and stream responses. The framework includes a Kubernetes operator for production deployment, a web UI built with Next.js and React for chatbot interaction and developer tooling, and a TypeScript client library. It also features built-in agent types like SimpleAgent, RetrieverAgent, SQLAgent, and a Tree-of-Thought agent, along with logic units for web search, browser automation, and API integration.
This repository provides a comprehensive toolkit for Chinese Llama-3 models, including base and instruction-tuned versions. It offers scripts for pre-training, fine-tuning, model merging, inference (via Hugging Face transformers, llama.cpp, Ollama, and OpenAI API-compatible server), and evaluation on standard benchmarks (CEval, CMMLU, MMLU, LongBench). The repository follows a modular architecture where each functionality is implemented in separate scripts under the scripts/ directory. Key entry points include training scripts (run_clm_pt_with_peft.py, run_clm_sft_with_peft.py), inference scripts (inference_hf.py, llama_cpp/chat.sh, openai_api_server.py), and evaluation scripts (ceval/eval.py, cmmlu/eval.py, etc.). The model supports LoRA adapters for efficient fine-tuning and provides merging utilities to combine base models with LoRA weights. The code demonstrates good practices in handling model quantization, PEFT integration, and multi-platform deployment.