PMetal is a comprehensive machine learning framework specifically optimized for Apple Silicon hardware. It provides a complete suite for LLM training, fine-tuning, inference, and deployment through multiple interfaces (CLI, TUI, GUI, Python SDK, and REST API). The framework leverages Apple-specific technologies like Metal GPU, ANE, and UltraFusion for performance optimization while supporting standard ML workflows including LoRA/QLoRA fine-tuning, knowledge distillation, model merging, and quantization to formats like GGUF. The architecture follows a modular workspace design with over 20 specialized crates handling different aspects of the ML lifecycle, unified through a job specification system and configuration framework.
This repository implements Speculative Speculative Decoding (SSD), a novel LLM inference algorithm that enables parallel execution of draft and target models on separate hardware. The system provides a complete inference engine with tensor parallelism support, optimized kernels (FlashInfer, CUDA graphs), and advanced KV caching techniques. It includes benchmarking tools to compare performance against baselines like SGLang and vLLM, and supports both synchronous and asynchronous speculative decoding modes. The codebase is well-structured with clear separation of concerns between model implementations, engine components, and utility functions.
This repository is a curated collection of learning materials for AI infrastructure, specifically focused on large language model (LLM) inference optimization. It contains detailed markdown documents covering core topics such as vLLM quickstart, PagedAttention, Prefix Caching, Speculative Decoding, Chunked-Prefills, and Prefill-Decode (PD) Disaggregation. Each lesson includes in-depth explanations, diagrams, code examples, and references to relevant papers and talks, making it a comprehensive resource for engineers and researchers seeking to understand and apply state-of-the-art LLM serving techniques. The content is organized as a series of study sessions, with each lesson building on previous ones. The repository does not contain any runnable application code or data pipelines; it is purely educational material. However, the code snippets provided (e.g., Python scripts for vLLM offline and online inference) are directly usable and demonstrate practical usage of the discussed technologies. The documentation is well-structured, with clear learning objectives, prerequisites, and supplementary resources. While the repository itself is not a software application, its value lies in the depth and clarity of its technical explanations. It serves as an excellent reference for anyone deploying or optimizing LLM inference systems, and the content is actively maintained with recent commits and issue tracking. The absence of reusable software components is expected given its educational nature, but the quality of the documentation is high, earning a strong quality score.
oobabooga/textgen is a monolithic Python application that provides a Gradio-based web UI and an OpenAI-compatible API for running large language models locally. It supports multiple inference backends (llama.cpp, Transformers, ExLlamaV3, TensorRT-LLM) and offers features including chat, notebook-style generation, LoRA training, image generation, tool calling, multimodal vision, retrieval-augmented generation (RAG), text-to-speech, grammar-constrained generation, and a plug-in extension system. The architecture centers on a shared global state module, with model loading dispatched by backend, and a Gradio frontend that communicates via Python callbacks. Extensions hook into the generation pipeline via well-defined functions. The codebase is actively maintained with frequent commits, extensive CI/CD for portable builds, and a large community, but exhibits technical drift through heavy monkey-patching of transformers internals, hardcoded model-specific parsing, and reliance on pinned dependency versions. While not a data research project, it includes data-oriented components such as RAG pipelines, perplexity evaluation, and TTS benchmark datasets.
KAITO (Kubernetes AI Toolchain Operator) is a production-grade Kubernetes operator that automates the full lifecycle of large language model (LLM) deployments. It reconciles custom resources—Workspace, InferenceSet, RAGEngine, MultiRoleInference, and ModelMirror—to provision GPU nodes, download or stream model weights from HuggingFace or Azure Blob, run inference servers (vLLM or HuggingFace Transformers), perform fine-tuning (LoRA/QLoRA), and manage RAG pipelines with vector stores (FAISS/Qdrant) and output guardrails. The operator supports multi-cloud node provisioning (Azure GPU provisioner, Karpenter, BYO) and features like prefill-decode disaggregation, model mirroring, and benchmark metrics collection. The system architecture is a collection of Go controllers using controller-runtime, with admission webhooks for CRD validation and a set of Python services for inference, tuning, and RAG. Key entry points: cmd/workspace/main.go (workspace controller), cmd/ragengine/main.go (RAG engine controller), presets/workspace/inference/vllm/inference_api.py (vLLM inference service), and presets/ragengine/main.py (RAG FastAPI service). Data flows: user creates a Workspace CR → controller creates NodeClaims, provisions GPU nodes, downloads/streams model weights → creates StatefulSet with inference service → exposes Service. For RAG: user creates RAGEngine CR → controller deploys vector store and inference service → user indexes documents via API → chat completions with retrieval augmentation. Notable engineering patterns include a plugin-based model registry (KaitoModelRegister), a generator pattern (WorkspaceGeneratorContext with chainable manifest modifiers), an abstract NodeProvisioner interface with multiple implementations, and a hot-reloadable guardrails system with Prometheus metrics. The codebase has extensive test coverage: unit tests with mock clients, integration tests for controllers, and end-to-end tests using Ginkgo that validate real cluster behavior. The project is actively maintained with frequent commits, CI/CD workflows, and versioned documentation.
This repository implements λ-RLM (Lambda Recursive Language Model), a framework for long-context reasoning that replaces free-form recursive code generation with a typed functional runtime grounded in λ-calculus. The system provides deterministic recursive decomposition, typed symbolic control flow, and bounded model calls at leaf subproblems. It features a modular architecture with pluggable LLM clients, sandboxed execution environments, and a benchmarking suite for comparison against standard RLM approaches. The core innovation is executing a pre-verified combinator chain (Φ) instead of open-ended LLM-driven while loops, guaranteeing termination and bounded cost.
Xinference is a production-grade model serving platform that provides OpenAI-compatible REST APIs for a wide range of AI models including LLMs, embeddings, rerank, image generation, video, audio/TTS, and world models. It supports multiple inference backends (vLLM, SGLang, transformers, MLX, llama.cpp) and can be deployed as a single process or a distributed cluster using an actor-based supervisor/worker architecture (xoscar). The system includes a web UI for model management, authentication/authorization with OAuth2 scopes, token-aware routing for load balancing, and monitoring capabilities. It bundles many third-party model implementations for turnkey deployment, which reduces setup friction but introduces maintenance overhead from vendored code. The architecture is a hybrid of a monolithic FastAPI server and a distributed actor system. Key entry points are CLI commands (xinference, xinference-supervisor, xinference-worker) and the REST API. The frontend is a Next.js application statically exported and served by the backend. The codebase demonstrates strong patterns in modular route registration, plugin-style model registries, and streaming response handling, but also carries significant technical debt from heavy vendoring, duplicated code across third-party integrations, and version-sensitive patches to upstream libraries like vLLM.
This repository is the official release hub for the MiniCPM family of compact large language models (1B to 8B parameters) from OpenBMB. It provides model weights, deployment scripts for 10+ inference backends (vLLM, SGLang, llama.cpp, Ollama, etc.), fine-tuning recipes for 5 frameworks (TRL, LLaMA-Factory, unsloth, etc.), quantization tools (AWQ, GPTQ, BNB), and application demos (survey generation, MCP tool calling, code interpreter). The repo is primarily a model card and cookbook collection rather than a single integrated application; it serves as a developer toolkit for deploying and customizing MiniCPM models in production and edge scenarios. The repository also includes research artifacts such as training datasets (UltraData suite) and technical reports, but the dominant character is applied machine learning engineering.
pysheeet is a comprehensive Python cheat sheet and reference repository. Although it is classified as an application because it contains a Flask web server and operational scripts, its dominant character is educational reference content: a Sphinx-built documentation site with runnable, tested code examples under `src/basic/`, covering core Python, asyncio, concurrency, networking, security, databases, C/C++ extensions, and more recent LLM/HPC topics. The repository also ships a Claude Code plugin that exposes the cheat sheets as AI skills. Beyond documentation, the repo contains real operational artifacts for distributed LLM work: Slurm/Docker launch scripts for vLLM, SGLang, and TensorRT-LLM, Megatron training recipes, NCCL broadcast benchmarks, and Ray-on-Slurm patterns. These scripts demonstrate practical HPC skills such as EFA/NCCL tuning, multi-node container orchestration, and profiling with nsys/VizTracer. The code quality is solid overall: examples are well-structured, many have pytest coverage, and the Flask serving layer is security-conscious. The main weaknesses are duplication across LLM vendor scripts, hardcoded cluster-specific paths, and documentation that occasionally references older Python or library versions, which creates moderate technical drift.
This repository is the engineering backbone for llamafile, a project by Mozilla.ai that packages large language model (LLM) inference engines into single, portable executables. It is primarily an application distribution and runtime platform, not a data research project. The core innovation is the use of Cosmopolitan Libc to create "Actually Portable Executables" (APEs) that run on Linux, macOS, Windows, and BSD without installation. The codebase integrates heavily patched versions of llama.cpp, whisper.cpp, stable-diffusion.cpp, and agent.cpp as submodules. It provides a unified CLI, TUI (chatbot), and HTTP server (OpenAI-compatible API) for interacting with these models. It features a sophisticated dynamic GPU backend loader (CUDA, ROCm, Vulkan, Metal), a custom high-performance CPU matrix multiplication library (TinyBLAS), integer quantized kernels (IQK), and a cross-platform sandboxing module (pledge/SECCOMP/Landlock). An agent framework (agentfile) allows for tool-calling workflows. The codebase demonstrates high engineering craftsmanship in specific areas, particularly the TinyBLAS CUDA replacement, the dynamic GPU backend abstraction, the state-machine syntax highlighting library, and the embedded readline replacement (Bestline). However, the project carries significant technical drift risk due to its heavy reliance on fast-moving upstream submodules and a complex, platform-specific build system. The patch management workflow is well-documented but inherently fragile.
This repository is the LLM Inference Handbook, a comprehensive technical documentation site built with Docusaurus. It serves as an educational resource for engineers building, optimizing, and deploying LLM inference systems. The site covers inference fundamentals, model preparation, interaction patterns, optimization techniques (batching, KV cache, prefix caching, speculative decoding), kernel optimization (FlashAttention, GPU architecture), and infrastructure operations. It includes interactive React components such as GPU memory calculators, KV cache simulators, batching visualizers, and GPU execution visualizers that demonstrate concepts through live demos. The codebase is well-structured with TypeScript, CSS modules, a custom design system (DLS), and integration with Mantine UI. While primarily a documentation site, the interactive components are self-contained and could be extracted for reuse in other educational contexts. The handbook is framework-agnostic but has a strong emphasis on the Modular MAX ecosystem.
PA Agent is a desktop application (PyQt6) that combines Al Brooks' price action theory with large language models (DeepSeek, MiMo, QClaw, etc.) to generate two-stage trading analysis: Stage 1 diagnoses market structure (cycle, pattern, direction), Stage 2 produces a trading decision (order type, entry, stop-loss, take-profit). The system ingests K-line data from multiple sources (MT5, TradingView, East Money, Tushare, AkShare) and provides real-time charting, AI reasoning streaming, decision tree visualization, and demo replay. It includes deterministic fallback nodes for data validation, direction voting, and pattern routing, reducing reliance on LLM output consistency. The architecture is modular with a data source abstraction, AI provider abstraction, event bus, incremental indicator computation, and a two-stage orchestration pipeline with retry and validation. While the codebase is well-structured and heavily tested (unit, integration, property-based), it suffers from complexity due to many provider connectors, hardcoded normalizer heuristics, and prompt engineering files that require ongoing maintenance.
monkeSearch is a macOS-native file search tool that uses a local LLM to parse natural language queries into structured search parameters, then executes Spotlight searches via PyObjC. The system breaks queries into file type indicators (with specificity detection), temporal expressions, and residual keywords, converting them to NSPredicates using UTIs for file type matching and date comparisons for temporal filtering. It features a clean separation between LLM-based query parsing (query_gen.py) and Spotlight execution (parser.py), with the former being particularly well-engineered for reuse. The tool is designed for offline, agentic use with no cloud dependencies, targeting low-resource environments.
llm-action is a comprehensive resource hub for large language model engineering that combines theoretical explanations with practical implementation code. It covers the full LLM lifecycle including data engineering, training (full fine-tuning and parameter-efficient methods like LoRA/QLoRA), model compression (quantization, pruning, distillation), inference serving, deployment, and evaluation. The repository contains both educational documentation and runnable code examples, with particular emphasis on Chinese-language resources and domestic AI hardware adaptation (especially Huawei Ascend NPUs). While it serves as a knowledge base, its inclusion of actual training scripts, inference servers, and utility tools makes it primarily application-oriented rather than purely research-focused.
Eko is a TypeScript framework for building production-ready AI agents that can automate workflows across browser, Node.js, and web extension environments. The core orchestration engine (Eko class) takes a natural language task, uses an LLM to generate a multi-agent workflow plan encoded as XML, then executes it with parallel/serial agent execution, replanning, and human-in-the-loop support. The system provides a clean abstraction over LLM providers (OpenAI, Anthropic, etc.) with retry and fallback, a ReAct tool-calling loop, and built-in browser automation agents that operate via screenshots or DOM labels. It also includes MCP client implementations (SSE, HTTP, stdio) for integrating external tools, and a chat agent for conversational interactions with tool delegation. The architecture is modular with separate npm packages: @eko-ai/eko-core (core engine), @eko-ai/eko-nodejs (Playwright-based browser agent), @eko-ai/eko-extension (Chrome extension agent), and @eko-ai/eko-web (web environment agent). The repository includes working examples (Chrome extension with side panel, Node.js automation using Playwright, and a React web app). The framework is designed for developers who want to automate complex, multi-step tasks using language models, with a focus on reliability, human oversight, and extensibility via custom agents and tools.
This repository is a research prototype for efficient LLM inference, specifically optimizing attention memory management via the vAttention technique (ASPLOS 2025). It contains three primary components: (1) a modified NVIDIA GPU kernel driver (nvidia-vattn-uvm-driver) that enables sub-2MB page sizes for Unified Virtual Memory (UVM), (2) a fork of the Sarathi-Serve LLM serving framework (sarathi-lean) integrated with vAttention, and (3) custom attention kernel implementations (pod_attn, vattention) using CUTLASS and FlashAttention backends. The system supports multiple attention backends (FlashAttention, FlashInfer, paged, variable-length) and includes extensive benchmarking and plotting scripts for reproducibility of research results. Architecturally, the user-space inference stack uses a scheduler (Orca or Sarathi), a cache engine (vLLM-style or vAttention), and model executors with tensor/pipeline parallelism. The vAttention cache engine replaces traditional paging with CUDA virtual memory allocation, decoupling virtual and physical memory for KV-cache. The custom NVIDIA UVM driver patch is required to support fine-grained page sizes. The repository also includes microbenchmarks for attention kernel performance and page size effects, as well as artifact scripts to reproduce figures from the associated paper. While the codebase demonstrates solid engineering for a research project (clean abstractions, registry pattern, modular design), it is explicitly not production-ready. Key limitations include strict version dependencies (PyTorch 2.3.0, CUDA 12.1), reliance on a modified kernel driver, and incomplete feature parity with production systems like vLLM. The inclusion of a fully vendored CUTLASS library and auto-generated NVIDIA driver headers adds significant size but provides reusable GPU primitives. The research scripts are functional but contain hardcoded paths and are tightly coupled to the experimental setup.
ExLlamaV2 is a high-performance inference library for running quantized large language models (LLMs) on consumer GPUs, supporting GPTQ and the custom EXL2 quantization formats. It provides a Python library with custom CUDA/C++ extensions for efficient attention, MLP, sampling, and KV cache management, along with advanced features like dynamic batching, paged attention, speculative decoding, LoRA, tensor parallelism, and multimodal (image/video) support. The library includes a WebSocket server for remote inference and a suite of example scripts demonstrating various use cases. While the codebase exhibits strong engineering with clean abstractions (e.g., module system, filter pattern, multi-backend attention) and highly optimized CUDA kernels, the repository has been archived in favor of ExLlamaV3, meaning no further updates or bug fixes are expected. This makes it a valuable reference for understanding quantized LLM inference internals but risky for new projects requiring ongoing support.