SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
This repository is the official companion code for the book *Build a Large Language Model (From Scratch)* by Sebastian Raschka (105k+ stars). It provides a step-by-step, educational implementation of GPT-2 and its modern variants (Llama 3, Qwen3, Gemma 3/4, Olmo3) entirely in PyTorch, organized by book chapters (2–7 plus appendices). The codebase is a mix of Jupyter notebooks for interactive learning and Python modules that form a reusable pip package (`llms-from-scratch`), accompanied by extensive CI/CD pipelines, unit tests, and performance analysis scripts. Architecturally, the core is a modular transformer library that includes tokenization (BPE, tiktoken, custom), multi-head attention with multiple variants (GQA, MLA, SWA, DSA, cross-layer KV sharing), feedforward blocks, normalization (LayerNorm, RMSNorm), and LoRA. The code supports pretraining from scratch, classification finetuning (spam, IMDB), instruction finetuning (with/without LoRA), and preference tuning (DPO). It can load pretrained weights from OpenAI, Hugging Face, and local checkpoints. The package exposes clean interfaces (`GPTModel`, `generate`, `train_model_simple`) and includes bonus materials for KV cache implementations, mixture-of-experts, memory estimators, and user interfaces (Chainlit). While primarily educational, the code exhibits production-quality patterns: isolated attention modules, configurable model architecture, well-structured training loops, and robust weight loading with shape validation. The repository is actively maintained with recent commits, cross-platform support (Linux, macOS, Windows), and a strong emphasis on reproducibility (fixed seeds, deterministic settings, environment definitions via pixi/uv/conda). Duplication exists across bonus notebooks and a few hardcoded paths, but the core package and test suite are well-coordinated.
LightGlue is a PyTorch library for local feature matching between image pairs. It implements a deep neural network that takes sparse keypoints and descriptors (extracted by models like SuperPoint, DISK, ALIKED, or SIFT) and outputs correspondences. The architecture uses stacked self- and cross-attention transformer layers with adaptive pruning (width and depth) for speed, achieving state-of-the-art accuracy at high throughput. The repository provides pretrained weights, a benchmark script, and a demo notebook. It is designed for inference only; training code is in a separate repository (glue-factory). The code is well-structured, with clean separation between feature extractors, the matcher, and utility functions. It supports FlashAttention and PyTorch compilation for performance.
FlagAttention is a library implementing memory-efficient attention operators in the Triton language. It provides optimized implementations of flash attention, piecewise attention, paged attention, and gated delta rule operations, designed for use in large language model training and inference. The library features modular architecture with separate implementations for each attention variant, comprehensive testing, and example usage patterns. It leverages Triton for GPU-accelerated computations while maintaining compatibility with PyTorch's autograd system.
This repository is a comprehensive collection of annotated PyTorch implementations of deep learning papers, spanning transformers, GANs, reinforcement learning, diffusion models, and more. Each implementation includes side-by-side notes explaining the code and underlying concepts, making it an educational resource for understanding and reproducing paper results. The codebase is organized by paper/model type, with standalone experiment scripts, reusable modules, and configuration-driven training loops. Key strengths include clean abstractions for transformer components, a flexible configs system, and extensive coverage of both foundational and modern architectures. However, the project has not seen commits since early 2024, some dependencies are outdated, and certain experiments require large GPU resources or external data not bundled in the repo. While individual components are well-crafted and reusable, the overall collection is best used as a reference and learning tool rather than a production framework.
This repository provides PyTorch implementations of Google Research's MetNet family of models for short-term weather forecasting, including MetNet, MetNet-2, MetNet-3, MetNetPV (photovoltaic), and MetNetSingleShot. The codebase is structured as a Python package with modular layers (ConvGRU, ConvLSTM, CoordConv, FiLM conditioning, MaxViT, partition attention, etc.) and data preprocessing utilities (Gaussian satellite merging, point-to-grid translation, MRMS normalization). The models are designed to ingest satellite imagery and radar data and output precipitation or other weather variables. The repository also includes pretrained weights hosted on HuggingFace Hub and a test suite covering layers, data utilities, and model forward/backward passes. While primarily an application library, it contains reusable data engineering components for geospatial data fusion.
This is a research-grade PyTorch implementation of Danijar Hafner's Dreamer v4 model-based reinforcement learning algorithm. The core artifact is the `dreamer4` package, dominated by a monolithic `dreamer4/dreamer4.py` (~3000 lines) that implements `VideoTokenizer`/`SpaceTimeTokenizer`, `DynamicsWorldModel`, `AudioTokenizer`, and `StateTokenizer`. These classes build causal space-time transformers with separate space/time attention blocks, latent flow-matching diffusion with shortcut consistency, BYOL-style tokenizer pre-training, PoPE attention, and a large zoo of experimental loss terms (latent autoregressive loss, decorrelation, orthogonality, SigReg, self-flow distillation, shortcut loss). `dreamer4/trainers.py` provides training loops (VideoTokenizerTrainer, BehaviorCloneTrainer, DreamTrainer, SimTrainer) and `dreamer4/cli.py` exposes CLI entry points for training and serving the world model. The repository is hybrid: primarily research code, but packaged and tested like a library. It ships a pyproject.toml with pinned dependencies, a parametrized test suite (tests/test_dreamer.py, test_e2e.py, test_env_wrapper.py), a browser-based Snake environment with a Flask-style server (`dreamer4/web_env/`) used for human demonstration collection and replay-buffer inspection, and environment wrappers in `dreamer4/env.py` that record agent experience to folders or memmap replay buffers. Root-level scripts are the actual experiment entry points, targeting standard benchmarks: HalfCheetah (imagination RL), CartPole (dynamics + RL), MovingMNIST (tokenizer + dynamics), and Snake (PPO with dynamics model). Craftsmanship is above typical research code: clean package layout, pytest coverage across many hyperparameter combinations, self-contained utilities (LossNormalizer, tensor pack/unpack helpers, with_seed decorator, memmap replay-buffer helpers), and HuggingFace Accelerate integration for distributed/mixed-precision training. However, the single-file core, 100+ hyperparameter flags on trainers, scattered hard-coded constants, sparse docstrings, and reliance on volatile niche packages (x-transformers, h-net-dynamic-chunking, PoPE_pytorch, adam-atan2-pytorch) make long-term stability and code reuse risky. The repository is recently committed and appears maintained, but it is explicitly beta-status experimental research code, not a production system.
This repository is the official code companion to the book '50 ML projects to understand LLMs' by Mike X Cohen. It contains over 50 Jupyter notebooks organized into chapters (2–7), each exploring a specific concept in large language model interpretability and internals. Topics span tokenization, embeddings, cosine similarity, softmax/logits, loss functions, sampling, hidden state analysis, effective dimensionality, logit lens, attention mechanisms, MLP projections, probing classifiers, and bias measurement. The code is purely educational and experimental—no production services, APIs, or deployable applications. All notebooks are designed to run on Google Colab with minimal setup, using standard libraries (PyTorch, Hugging Face Transformers, scikit-learn, numpy, matplotlib) and pretrained models (GPT-2, BERT, RoBERTa, Pythia). Each project has a 'helper' notebook (incomplete code for students) and a 'solution' notebook (complete reference implementation). The repository is a valuable resource for learning mechanistic interpretability and hands-on LLM analysis, but it lacks modularization, dependency pinning, and test coverage, making it unsuitable for direct reuse in production pipelines.
This repository is a research prototype for efficient LLM inference, specifically optimizing attention memory management via the vAttention technique (ASPLOS 2025). It contains three primary components: (1) a modified NVIDIA GPU kernel driver (nvidia-vattn-uvm-driver) that enables sub-2MB page sizes for Unified Virtual Memory (UVM), (2) a fork of the Sarathi-Serve LLM serving framework (sarathi-lean) integrated with vAttention, and (3) custom attention kernel implementations (pod_attn, vattention) using CUTLASS and FlashAttention backends. The system supports multiple attention backends (FlashAttention, FlashInfer, paged, variable-length) and includes extensive benchmarking and plotting scripts for reproducibility of research results. Architecturally, the user-space inference stack uses a scheduler (Orca or Sarathi), a cache engine (vLLM-style or vAttention), and model executors with tensor/pipeline parallelism. The vAttention cache engine replaces traditional paging with CUDA virtual memory allocation, decoupling virtual and physical memory for KV-cache. The custom NVIDIA UVM driver patch is required to support fine-grained page sizes. The repository also includes microbenchmarks for attention kernel performance and page size effects, as well as artifact scripts to reproduce figures from the associated paper. While the codebase demonstrates solid engineering for a research project (clean abstractions, registry pattern, modular design), it is explicitly not production-ready. Key limitations include strict version dependencies (PyTorch 2.3.0, CUDA 12.1), reliance on a modified kernel driver, and incomplete feature parity with production systems like vLLM. The inclusion of a fully vendored CUTLASS library and auto-generated NVIDIA driver headers adds significant size but provides reusable GPU primitives. The research scripts are functional but contain hardcoded paths and are tightly coupled to the experimental setup.