This repository is the official companion code for the book *Build a Large Language Model (From Scratch)* by Sebastian Raschka (105k+ stars). It provides a step-by-step, educational implementation of GPT-2 and its modern variants (Llama 3, Qwen3, Gemma 3/4, Olmo3) entirely in PyTorch, organized by book chapters (2–7 plus appendices). The codebase is a mix of Jupyter notebooks for interactive learning and Python modules that form a reusable pip package (`llms-from-scratch`), accompanied by extensive CI/CD pipelines, unit tests, and performance analysis scripts. Architecturally, the core is a modular transformer library that includes tokenization (BPE, tiktoken, custom), multi-head attention with multiple variants (GQA, MLA, SWA, DSA, cross-layer KV sharing), feedforward blocks, normalization (LayerNorm, RMSNorm), and LoRA. The code supports pretraining from scratch, classification finetuning (spam, IMDB), instruction finetuning (with/without LoRA), and preference tuning (DPO). It can load pretrained weights from OpenAI, Hugging Face, and local checkpoints. The package exposes clean interfaces (`GPTModel`, `generate`, `train_model_simple`) and includes bonus materials for KV cache implementations, mixture-of-experts, memory estimators, and user interfaces (Chainlit). While primarily educational, the code exhibits production-quality patterns: isolated attention modules, configurable model architecture, well-structured training loops, and robust weight loading with shape validation. The repository is actively maintained with recent commits, cross-platform support (Linux, macOS, Windows), and a strong emphasis on reproducibility (fixed seeds, deterministic settings, environment definitions via pixi/uv/conda). Duplication exists across bonus notebooks and a few hardcoded paths, but the core package and test suite are well-coordinated.
This is the official PyTorch tutorials repository, a Sphinx-Gallery documentation site that publishes to pytorch.org/tutorials. It contains hundreds of executable tutorial scripts covering beginner to advanced PyTorch topics (neural networks, distributed training, torch.compile, performance profiling, custom operators, etc.). The repository is not a standalone application but a documentation generator with sophisticated CI/CD pipelines for building, sharding, and testing tutorials across multiple GPU configurations. It includes extensive build tooling (Docker, Jenkins, GitHub Actions), linters, spell checking, link validation, and redirect management. The code within tutorial files is pedagogical and not production-ready; however, several utility scripts and patterns are reusable for documentation projects and PyTorch extension development.
Deep Java Library (DJL) is a comprehensive, engine-agnostic deep learning framework for Java, providing a unified API for model loading, training, inference, and deployment across multiple backends (PyTorch, TensorFlow, MXNet, ONNX Runtime, XGBoost, LightGBM, and a Rust-based engine). Its architecture is modular: a core API defines abstractions like `NDArray`, `NDManager`, `Model`, `Predictor`, `Trainer`, and `Translator`; engine-specific implementations adapt these to native libraries; model zoos and repositories supply pre-trained models; and extensions add capabilities for generative AI, audio, timeseries, Spark integration, and more. The codebase is well-organized with extensive use of the builder pattern, ServiceLoader-based plugin discovery, and a rich set of training utilities (losses, optimizers, trackers, evaluators, listeners). It supports both training and inference workflows and includes runnable examples for CV, NLP, audio, and tabular tasks. While the project is actively maintained with frequent commits and CI, it exhibits medium technical drift due to some outdated engine dependencies (e.g., MXNet) and the need to synchronize multiple native library versions.
Protenix is a production-oriented PyTorch implementation of AlphaFold 3 for biomolecular structure prediction, maintained by ByteDance. It is best classified as an application rather than a pure research codebase: it ships a `protenix` CLI with commands for prediction, input conversion, MSA/template/RNA-MSA search, and data preparation, plus training/fine-tuning entry points, batch inference runners, Docker deployment, web-service helpers, and PyPI packaging. The code ingests PDB/CIF, FASTA, SMILES/SDF, and JSON descriptors covering proteins, DNA, RNA, ligands, ions, modified residues, covalent bonds, and contact/pocket constraints, then outputs predicted 3D CIF structures with confidence metrics such as pLDDT, PAE, pTM, and ipTM.
nnsight is a Python library for interpreting and intervening in neural network models during execution. It provides a tracing API that allows users to access and modify internal activations, compute gradients, and perform interventions like activation patching and steering. The system uses a deferred execution model with thread-based synchronization to run user intervention code alongside model forward passes. Key features include support for Hugging Face models, vLLM integration for high-performance inference, remote execution via NDIF infrastructure, and persistent model editing. The architecture centers around Envoy wrappers for modules, a tracer/interleaver system for coordinating execution, and a batching mechanism for efficient multi-input processing.
Tutel is a high-performance library for Mixture-of-Experts (MoE) in PyTorch, focused on both training and inference of large-scale sparsely-gated models. The codebase is organized into distinct layers: the core `MOELayer` configuration (gating, experts, dispatch), custom NCCL-based distributed communication primitives (`net.py`), JIT-compiled CUDA/ROCm kernels for routing and sparse operations, and utility modules for checkpoint resharding across different distributed topologies. Heavily exploited features include low-precision quantized inference (FP4, FP8, block quantization), fused Expert comunication via all-to-all collectives, and a experimental auto-paritioning solver for adaptive parallelism. The repository is dominated by software engineering artifacts (Python library design, C++/CUDA/ROCm extensions and binaries, structured examples) rather than research notebooks or analytic pipeline, but its origins include published papers (MLSys 2023) and active development for contemporary models such as DeepSeek, Qwen3, and Kimi.
onnx2tf is a production-ready ONNX to TensorFlow/TFLite conversion framework that employs a sophisticated pass-based optimization pipeline. The system converts ONNX models through preprocessing, lowering to an intermediate representation (ModelIR), applying 50+ optimization passes (layout transformation, operator fusion, quantization), and exporting to target formats including TFLite, SavedModel, PyTorch nn.Module, and TorchScript. Its architecture features clean separation between operator implementations, IR transformation passes, and backend-specific exporters, with the flatbuffer_direct backend providing 100x+ faster conversion by bypassing TensorFlow graph construction. The framework includes comprehensive accuracy validation tools, dynamic shape handling, and model surgery capabilities for intermediate layer extraction.
KerasHub is a production-grade Keras 3 library providing pretrained model architectures, layers, tokenizers, metrics, and sampling utilities for NLP, vision, multimodal, and audio tasks. The codebase is organized around a consistent Backbone → Task → Preprocessor pattern, where each model family (e.g., BERT, Gemma, ViT, Whisper) defines a backbone, a preprocessor (tokenizer + optional media converter), and task-specific heads (CausalLM, ImageClassifier, etc.). All models expose a unified `from_preset()` API for loading pretrained weights hosted on Kaggle Models, and `save_to_preset()` for export. The library includes extensive checkpoint conversion tooling from HuggingFace and TIMM, export to HuggingFace format and OpenVINO, and integration with vLLM for serving. While predominantly an application library, it has strong data/research character through its pretrained weight infrastructure, benchmark scripts, and research-oriented architectures (e.g., FNet, RWKV7, BLIP-2).
AlbumentationsX is a production-grade Python library for fast, flexible image augmentation in deep learning and computer vision pipelines. It provides a comprehensive catalog of 70+ transforms covering pixel-level (blur, noise, color, weather effects), geometric (flip, rotate, crop, resize, distortion, perspective), dropout (coarse, grid, mask, pixel), mixing (mosaic, copy-paste, overlay), and domain adaptation (FDA, histogram matching, stain normalization). The library supports 2D and 3D data, including images, masks, bounding boxes (HBB and OBB), keypoints, and volumes, with seamless integration into PyTorch via tensor transforms. Its architecture is built on a clean inheritance hierarchy (BasicTransform → DualTransform/ImageOnlyTransform/Transform3D) with Pydantic-based parameter validation, serialization to JSON/YAML, and deterministic replay via ReplayCompose. The codebase is backed by an extensive test suite (property-based, regression, contract, and golden vector tests), ASV benchmarks, and a sophisticated CI/CD system with path-aware planning, performance budgets, and legal integrity checks. Telemetry (opt-out) collects anonymous usage statistics to guide development. While primarily a software library, it incorporates research-level transforms (e.g., PlanckianJitter, HEStain, dithering) and is used in both academic and production settings.
Tangermeme is a PyTorch library for post-training interpretability of sequence-based deep learning models in regulatory genomics. It provides a comprehensive suite of functions for analyzing what a trained model has learned about cis-regulatory elements, including attribution methods (DeepLIFT/SHAP, saturation mutagenesis, PISA), sequence perturbation experiments (marginalization, ablation, variant effect, spacing analysis), motif discovery via seqlet calling (recursive and TF-MoDISco-style), and sequence design (greedy/beam search substitution). The library is designed to be assumption-free, extensible to any alphabet, and works with any PyTorch model that operates on one-hot encoded sequences. Key entry points are the `deep_lift_shap`, `predict`, `ersatz`, `marginalize`, `ablate`, `seqlet`, `annotate`, and `design` subpackages, all exposed through a consistent API that accepts a `func=` argument for composability. The library includes a built-in Claude Code skill installer for AI-assisted development, extensive unit tests with toy models, and documentation with tutorials and vignettes. Tangermeme is actively developed (semantic versioning, regular releases) and targets bioinformaticians working with models like BPNet, Enformer, Beluga, or ChromBPNet.
AudioX-Turbo is a unified framework for efficient anything-to-audio generation (text, video, audio conditions → audio) using Distribution Matching Distillation (DMD) adapted to flow matching and adversarial training. It supports six core tasks: Text-to-Audio, Text-to-Music, Video-to-Audio, Video-to-Music, Text+Video-to-Audio, and Text+Video-to-Music. The system includes training scripts, an interactive Gradio demo, and a modular architecture with interchangeable components (attention, bottlenecks, etc.) via a factory pattern. Key innovations enable high-quality 4-step audio generation, reducing computational cost by up to 25× compared to multi-step diffusion models.
Netron is a cross-platform neural network model visualizer that supports a wide range of deep learning frameworks (ONNX, TensorFlow, PyTorch, CoreML, etc.). It operates as an Electron desktop application, a web app, and a Python-based CLI/server. The architecture is modular: each model format is handled by a dedicated `ModelFactory` class implementing `match()` and `open()` methods, which parse binary or protobuf/flatbuffers serialized models into a uniform internal graph representation (Model -> Graph -> Node -> Value/Tensor). The application provides interactive graph visualization with SVG rendering, attribute inspection, and export capabilities. While primarily an application, it contains substantial data-processing code for format parsing, tensor decoding, and schema-driven metadata handling. The codebase is well-structured with consistent patterns, though it exhibits some duplication across format parsers and experimental/partial support for minority formats. The Python server (`server.py`) and the core binary parsing libraries (`base.js`, `protobuf.js`, `flatbuffers.js`) are independently reusable.
Silero VAD is a pre-trained enterprise-grade Voice Activity Detector library that provides speech detection capabilities for audio signals. The system consists of a core PyTorch model (also available in ONNX format) with Python utilities for audio processing, speech timestamp extraction, and streaming applications. It supports 8kHz and 16kHz sampling rates and includes a VADIterator class for real-time processing. The repository offers multi-language examples (C++, C#, Go, Haskell, Java, Rust) demonstrating integration, along with tools for model tuning and sequence-based ONNX optimization for improved performance on long audio files.
Cog is an open-source CLI and runtime for packaging machine learning models into production-ready Docker containers. It automatically generates HTTP inference servers with OpenAPI schema, handles GPU/CUDA/cuDNN compatibility, and provides a full weight management pipeline for large model artifacts. The project consists of a Go CLI (build, push, serve, predict, doctor, weights management), a Rust/PyO3 prediction server (coglet) with slot-based concurrency, a Python SDK for defining model interfaces (BaseRunner, Input, streaming), and a TypeScript playground UI. Key workflows include building OCI images from cog.yaml, pushing to registries, serving models via HTTP with SSE streaming, and managing model weights via content-addressed storage and lockfiles. The architecture is multi-language but cleanly separated, with well-defined protocols between components.
LightGlue is a PyTorch library for local feature matching between image pairs. It implements a deep neural network that takes sparse keypoints and descriptors (extracted by models like SuperPoint, DISK, ALIKED, or SIFT) and outputs correspondences. The architecture uses stacked self- and cross-attention transformer layers with adaptive pruning (width and depth) for speed, achieving state-of-the-art accuracy at high throughput. The repository provides pretrained weights, a benchmark script, and a demo notebook. It is designed for inference only; training code is in a separate repository (glue-factory). The code is well-structured, with clean separation between feature extractors, the matcher, and utility functions. It supports FlashAttention and PyTorch compilation for performance.
This repository, NVlabs/tiny-cuda-nn, is a high-performance C++/CUDA library and PyTorch extension for training and querying neural networks with a focus on extreme GPU efficiency. It provides a modular, JSON-configurable framework of specialized components: input encodings (multiresolution hash grids, spherical harmonics, frequency, etc.), network architectures (fully fused MLP using Tensor Cores, CUTLASS-based MLP), loss functions, and optimizers (Adam, Shampoo, Novograd, etc.). The library supports runtime JIT compilation to fuse forward, backward, and loss kernels into single CUDA kernels, yielding up to 5x speedups. It is the engine behind NVIDIA's Instant Neural Graphics Primitives (Instant NGP) and is widely used in NeRF and neural rendering research. The codebase is production‑quality, with extensive use of templates, GPU memory arenas, CUDA graphs, multi‑stream parallelism, and comprehensive CI across many CUDA versions and GPU architectures. Although primarily a library, it includes sample applications and benchmarks that demonstrate its use in image overfitting and throughput measurement. PyTorch bindings via pybind11 make it accessible for Python‑based research workflows.
DeePMD-kit is a comprehensive, production-grade deep learning package for many-body interatomic potential energy representation and molecular dynamics simulation. It provides a unified framework for training and deploying machine learning interatomic potentials (MLIPs) across multiple backends: TensorFlow, PyTorch, JAX, and PaddlePaddle, with a backend-agnostic core (dpmodel) that ensures numerical consistency. The package supports a wide range of descriptor architectures (SeA, SeR, SeT, DPA1/2/3/4, attention-based, hybrid) and fitting targets (energy, force, virial, dipole, polarizability, density of states, spin, property). It integrates tightly with LAMMPS via custom pair styles and plugins, and also supports ASE, i-PI, GROMACS, and other MD engines. The codebase is highly modular, with a plugin registration system for models, descriptors, and losses, and offers extensive serialization capabilities for model export and compression. Pretrained models (DPA-3, DPA-4) are available for direct download and fine-tuning. The project is actively maintained with a large community, comprehensive test suites, and continuous integration across CPU and GPU platforms.
ComfyUI is a modular, node-based AI engine for content creation, primarily focused on generative image, video, audio, and 3D model inference. It provides a web server (aiohttp) with WebSocket and REST APIs, a visual workflow editor, and an extensible plugin system for custom nodes. The architecture is a monolith with a clear separation of concerns: core model loading and sampling (comfy/), execution engine (comfy_execution/), asset management (app/assets/), external API integrations (comfy_api_nodes/), and a public API for defining nodes (comfy_api/). It supports a vast range of open-source diffusion models (Flux, SD3, Wan, Hunyuan, etc.) and offers advanced features like dynamic VRAM management, quantization, multi-GPU support, and experimental guidance methods. The system operates as a local-first application but also integrates with cloud services via partner API nodes. Key entry points include main.py (startup), server.py (PromptServer), and execution.py (PromptExecutor). The codebase demonstrates high craftsmanship in its weight adapter hierarchy (LoRA, LoHa, OFT), patcher extension system for model hooks, and asset management services with database-backed metadata extraction and cursor-based pagination. However, it also contains experimental and deprecated nodes, some code duplication across model implementations, and a reliance on external dependencies that may drift over time.
Qlib is an open-source, AI-oriented quantitative investment platform developed by Microsoft. It covers the full pipeline for quantitative research and trading: data ingestion and feature engineering (Alpha158, Alpha360, high-frequency), model training (traditional ML like LightGBM/XGBoost, deep learning in PyTorch including LSTM, GRU, Transformer, GATs, and reinforcement learning), backtesting with realistic costs and slippage, portfolio optimization, and online serving for production-like deployment. The platform is highly modular and config-driven, with a core library (qlib) and a separate contrib layer for community extensions. It includes a CLI (qrun) for running full workflows from YAML configs, extensive experiment tracking via MLflow, and a rich set of evaluation metrics. While primarily a library/application, it also functions as a research benchmark suite with many example models and data collectors for global markets. The codebase is active and well-maintained, though some components (e.g., TFT model, older RL code) show signs of technical drift due to dependency version locks and deprecated patterns.
TimesFM is a pretrained decoder-only transformer foundation model for time-series forecasting, developed by Google Research. The repository provides multiple model versions (v1.0, v2.0, v2.5, v3.0) with implementations in PyTorch, Flax/JAX, and MLX (Apple Silicon). It supports zero-shot univariate and multivariate forecasting with covariates, quantile outputs, and various preprocessing options (RevIN normalization, linear detrending, symmetric averaging). The codebase includes a high-level forecaster API, CLI scripts for CSV forecasting and system checks, fine-tuning examples (LoRA/DoRA), and comprehensive benchmarks (GIFT-Eval, TIME, FEV). While primarily an application library, it also contains research-oriented components like benchmark runners and notebooks, making it a hybrid project with a strong application focus.