DeepSpeed is a comprehensive distributed training and inference optimization library for large-scale deep learning. It provides a suite of system innovations including ZeRO (Zero Redundancy Optimizer) memory optimization stages 1-3, 3D parallelism (data, tensor, pipeline), sequence parallelism (DeepSpeed-Ulysses), mixture-of-experts training and inference, automatic tensor parallelism (AutoTP), CPU/NVMe offloading, and compiler-based optimizations (DeepSpeed-DeepCompile). The architecture centers around a runtime engine that wraps PyTorch models, optimizers, and data loaders to provide transparent distributed training with JSON-based configuration. Key entry points include `deepspeed.initialize()` for training, `deepspeed.init_inference()` for inference, and CLI launchers (`deepspeed`, `ds_report`). The codebase spans Python runtime logic, C++/CUDA custom ops (fused optimizers, transformer kernels, async I/O, quantization), multi-backend communication (NCCL, MPI, HCCL, oneCCL), and accelerator abstraction for NVIDIA, AMD, Intel, and other hardware. Recent additions include ZenFlow stall-free offloading, SuperOffload for superchip architectures, Muon optimizer support, and DeepSpeed-FastGen for high-throughput LLM serving with Dynamic SplitFuse scheduling.
llmfit is a comprehensive LLM model analysis and recommendation system that matches models to local hardware specifications. It detects system capabilities (CPU, RAM, GPU/VRAM), analyzes model requirements across multiple dimensions (memory fit, speed, quality, context), and provides recommendations for optimal model selection. The system features multiple interfaces including a terminal user interface (TUI) as the default mode, command-line interface, embedded web dashboard, and REST API. It supports various inference backends (Ollama, llama.cpp, MLX, Docker Model Runner, etc.), includes benchmarking capabilities to measure real-world performance, and enables users to share benchmark results with the community. The architecture follows a modular Rust workspace design with llmfit-core as the shared library containing hardware detection, model analysis, and provider abstractions, while interface-specific concerns are separated into llmfit-tui, llmfit-desktop, llmfit-web, and llmfit-python packages.
TurboQuant Model is a Python library and CLI tool for near-lossless LLM weight quantization, implementing the TurboQuant algorithm from Zandieh et al. (2025). It provides a drop-in replacement for nn.Linear (TurboQuantLinear) that stores weights in packed 4-bit indices and dequantizes on-the-fly during matrix multiplication using fused GPU kernels (CuTile, Triton, Metal). The library supports single-pass 4-bit, multi-pass residual (e.g., 4+4), polar decomposition, and hash-based compression, achieving 4x compression vs bf16 with KLD as low as 0.002 nats. It includes a CLI for quantizing, evaluating, generating, and benchmarking HuggingFace models, with CPU offload for residual passes and advanced techniques like norm compression and entropy coding. The codebase is well-structured with comprehensive tests and benchmarks, though it targets cutting-edge CUDA 13.1+ and may have limited hardware compatibility.
ColossalAI is a comprehensive distributed training and inference framework for large-scale deep learning models, providing multiple parallelism strategies (data, tensor, pipeline, sequence, mixture-of-experts), ZeRO memory optimization, automatic parallelization via FX tracing, and high-performance custom CUDA/Triton kernels. The architecture is library-based with a plugin-style booster for orchestrating training and inference, a shardformer for model-level parallelism policies, and a legacy subsystem for older APIs. It includes application-level tools: ColossalChat for RLHF (PPO, DPO, GRPO), ColossalQA for retrieval-augmented question answering, ColossalEval for LLM evaluation with GPT judges, and Colossal-LLaMA for domain-specific LLM training. The codebase is actively maintained with recent commits, but has notable technical drift due to outdated dependency pins in some examples and a large legacy namespace that coexists with newer implementations.
This repository is the LLM Inference Handbook, a comprehensive technical documentation site built with Docusaurus. It serves as an educational resource for engineers building, optimizing, and deploying LLM inference systems. The site covers inference fundamentals, model preparation, interaction patterns, optimization techniques (batching, KV cache, prefix caching, speculative decoding), kernel optimization (FlashAttention, GPU architecture), and infrastructure operations. It includes interactive React components such as GPU memory calculators, KV cache simulators, batching visualizers, and GPU execution visualizers that demonstrate concepts through live demos. The codebase is well-structured with TypeScript, CSS modules, a custom design system (DLS), and integration with Mantine UI. While primarily a documentation site, the interactive components are self-contained and could be extracted for reuse in other educational contexts. The handbook is framework-agnostic but has a strong emphasis on the Modular MAX ecosystem.
DART (Detect Anything in Real Time) is a training-free framework that converts SAM3 into a real-time multi-class open-vocabulary detector. In practice, the codebase is an inference-optimization and deployment framework: it exports SAM3 backbone components to ONNX/TensorRT, applies FP16/INT8 quantization, builds optimized TRT engines, and runs pipelined video inference with CUDA graphs and torch.compile. It also includes an LLM-driven interactive agent (Qwen via OpenAI-compatible API) that performs iterative segmentation through tool calls (segment_phrase, examine_each_mask, select_masks_and_return), plus video dataset handling, grounding/mask export utilities, and COCO-style evaluation scripts. The repository is a hybrid: fundamentally an application/framework for real-time detection, it carries strong research-code characteristics — calibration caches (calib/*.json), ad-hoc experiment scripts with hardcoded checkpoints (sam3.pt, backbone.onnx), and iterative debugging comments ('Previous test showed...'). The architecture centers on ONNX export of SAM3 blocks (EfficientViT backbone, text encoder, prompt encoder, mask predictor, pixel decoder), TRT engine building, and a quantization layer (dartf/quant/) with experimental plugins tightly coupled to specific ONNX graph patterns. A media production pipeline (dartf/media/) builds explainer videos with automated slide rendering and music mixing. The codebase shows signs of being an experimental workspace rather than a maintained library: hardcoded paths in evaluation configs, version-sensitive TensorRT API usage, a fragile ONNX export pipeline depending on onnxsim subprocess calls, and potential drift between ONNX-exported blocks and real backbone behavior. The 'training-free' claim is partially contradicted by the presence of distillation training code, and the package name dart-detect vs module sam3 mismatch may cause import confusion. Despite these issues, several well-isolated components — an abstract prediction-file evaluator, a TensorRT layer analysis script, and an LLM agent client — could be extracted for reuse in other vision or agent projects.