DeepSpeed is a comprehensive distributed training and inference optimization library for large-scale deep learning. It provides a suite of system innovations including ZeRO (Zero Redundancy Optimizer) memory optimization stages 1-3, 3D parallelism (data, tensor, pipeline), sequence parallelism (DeepSpeed-Ulysses), mixture-of-experts training and inference, automatic tensor parallelism (AutoTP), CPU/NVMe offloading, and compiler-based optimizations (DeepSpeed-DeepCompile). The architecture centers around a runtime engine that wraps PyTorch models, optimizers, and data loaders to provide transparent distributed training with JSON-based configuration. Key entry points include `deepspeed.initialize()` for training, `deepspeed.init_inference()` for inference, and CLI launchers (`deepspeed`, `ds_report`). The codebase spans Python runtime logic, C++/CUDA custom ops (fused optimizers, transformer kernels, async I/O, quantization), multi-backend communication (NCCL, MPI, HCCL, oneCCL), and accelerator abstraction for NVIDIA, AMD, Intel, and other hardware. Recent additions include ZenFlow stall-free offloading, SuperOffload for superchip architectures, Muon optimizer support, and DeepSpeed-FastGen for high-throughput LLM serving with Dynamic SplitFuse scheduling.
FastChat is a production-grade distributed serving platform for large language models that powers Chatbot Arena (lmarena.ai), handling 10M+ requests across 70+ models. The architecture centers on a controller that manages model worker registration, heartbeats, and request dispatching; model workers host models via multiple backends (HuggingFace transformers, vLLM, LightLLM, MLX, ExLlamaV2, xFasterTransformer, GPTQ, AWQ, dash-infer); an OpenAI-compatible RESTful API server routes chat/completion/embedding requests; and Gradio web UIs provide single-model chat and side-by-side battle interfaces. The codebase includes training pipelines for Vicuna/FastChat-T5 with FSDP/DeepSpeed/LoRA/QLoRA support, an MT-bench evaluation framework with LLM-as-a-judge (GPT-4), and data processing utilities for ShareGPT conversation cleaning. Key strengths: clean model adapter pattern for adding new architectures, extensible conversation template system, robust streaming API implementation, and well-integrated quantization backends. The system demonstrates non-obvious engineering in distributed inference coordination, attention optimization monkey patches, and multi-backend model loading abstraction.
PMetal is a comprehensive machine learning framework specifically optimized for Apple Silicon hardware. It provides a complete suite for LLM training, fine-tuning, inference, and deployment through multiple interfaces (CLI, TUI, GUI, Python SDK, and REST API). The framework leverages Apple-specific technologies like Metal GPU, ANE, and UltraFusion for performance optimization while supporting standard ML workflows including LoRA/QLoRA fine-tuning, knowledge distillation, model merging, and quantization to formats like GGUF. The architecture follows a modular workspace design with over 20 specialized crates handling different aspects of the ML lifecycle, unified through a job specification system and configuration framework.
This repository accompanies a Korean-language book on practical LLM application development, containing hands-on code examples organized by chapter. It demonstrates the full LLM lifecycle including foundational concepts (transformers), model optimization (quantization, PEFT), application building (RAG, agents, multimodal systems), and deployment techniques. The primary value lies in its educational notebooks that show how to implement LLM-powered solutions for tasks like Text2SQL, retrieval-augmented generation, and AI agents using frameworks such as LlamaIndex, AutoGen, and vLLM. While designed for Google Colab execution, the code illustrates reusable patterns for LLM engineering.