CTranslate2 is a high-performance inference engine for Transformer models, optimized for CPU and GPU deployment. It supports a wide range of architectures including encoder-decoder (BART, T5, NLLB, M2M-100, Whisper), decoder-only (LLaMA, GPT-2, Mistral, Gemma, Qwen), and encoder-only (BERT, DistilBERT, XLM-RoBERTa) models. The system provides a C++ library with Python bindings, a command-line tool (ct2-translator), and converters from popular frameworks like HuggingFace Transformers, Fairseq, Marian, OpenNMT-py/tf, and OPUS-MT. Key features include quantized inference (INT8/INT16/FP16/BF16/AWQ), flash attention, multi-GPU tensor parallelism, streaming generation, and token-level penalties. The architecture follows a modular design with device-agnostic operations, layer abstractions, and a model registry that allows plugging in new architectures without modifying the core engine. The codebase is actively maintained with frequent releases, extensive test coverage (C++ and Python), and CI pipelines for multiple platforms and backends.
This repository implements generative recommendation models, primarily HSTU (Hierarchical Sequential Transduction Units) and DLRM-v3, for large-scale sequential recommendation. It is a hybrid codebase combining research experimentation (ICML'24 paper on generative recommenders) with production-grade inference benchmarking (MLPerf loadgen integration). The code provides custom high-performance CUDA and Triton kernels for flash attention, jagged tensor operations, and layer normalization, targeting modern NVIDIA GPUs (Hopper/Blackwell). It includes training pipelines with gin configuration for datasets like MovieLens, Amazon Books, KuaiRand, and synthetic streaming data up to 100B items. The repository also contains a full MLPerf inference benchmark suite with vendored loadgen library and end-to-end TorchScript tracing for deployment. While the core focus is research on scaling laws and architecture innovations for sequential recommenders, the ops and modules are engineered to production standards with multiple kernel backends and rigorous tests.
nanochat is a minimal, hackable LLM training and inference framework focused on reproducing GPT-2 performance quickly on 8×H100 GPUs. It provides an end-to-end pipeline from tokenizer training and dataset preparation through pretraining, supervised fine-tuning, reinforcement learning (GRPO/REINFORCE), and a chat CLI. The core is a highly optimized GPT model with advanced features including a custom MuonAdamW hybrid optimizer, FP8 training with tensorwise dynamic scaling, unified Flash Attention 3/SDPA interface, sliding window attention, value embeddings, and BOS-aligned best-fit packing dataloader. Hyperparameters are automatically derived from scaling laws based on model depth, enabling compute-optimal training. The codebase is research-oriented, with extensive experiment logging, leaderboard tracking, and scaling law analysis notebooks. While primarily an application (CLI-driven training/evaluation framework), it has strong data_research elements including dataset pipelines, evaluation tasks (CORE, MMLU, GSM8K, etc.), and analytical notebooks.
Autoresearch is a self-contained autonomous ML experimentation framework created by Andrej Karpathy. Its core premise is that an AI agent (e.g., Claude or Codex) modifies a single training script (`train.py`), runs a fixed 5-minute training experiment on a single GPU, evaluates validation bits-per-byte (BPB), and keeps or discards changes based on whether the metric improved — all without human intervention. The human programs the agent via `program.md`, which serves as a lightweight instruction set for the experiment loop. The codebase has two fixed files and one editable file. `prepare.py` handles data downloading (parquet shards from HuggingFace), BPE tokenizer training (via rustbpe), a custom BOS-aligned packed dataloader with best-fit document packing (zero padding), and a fixed BPB evaluation harness. `train.py` contains a GPT model implementation with RoPE, sliding window attention (SSSL pattern), ResFormer value embeddings, RMS normalization, and a hybrid Muon+AdamW optimizer with per-parameter-type learning rate scheduling. The model uses `torch.compile` and runs in bf16 with Flash Attention 3. The `analysis.ipynb` notebook provides post-hoc visualization of experiment results from `results.tsv`. The project is explicitly designed for a single NVIDIA GPU (H100 tested), requires no distributed training infrastructure, and pins all dependencies including PyTorch 2.9.1 with CUDA 12.8. The architecture is deliberately minimal — the entire research loop runs in two Python files with roughly 1,600 lines of non-comment code total.
Text Embeddings Inference (TEI) is a production-grade inference server for open-source text embedding models, built in Rust with multiple backend implementations (Candle, Python, ONNX). The system exposes both HTTP (REST) and gRPC APIs supporting dense and sparse embeddings, reranking, sequence classification, and OpenAI-compatible endpoints. It features dynamic batching with cumulative sequence length management, flash attention optimization across CUDA compute capabilities, and comprehensive observability with Prometheus metrics and OpenTelemetry tracing. The architecture cleanly separates the router layer (HTTP/gRPC handling) from the core inference pipeline (tokenization, batching, backend dispatch) via the `Backend` trait, allowing seamless integration of different model backends and hardware targets including CPU, CUDA, ROCm, Intel, and ARM64. The codebase demonstrates strong engineering practices: modular neural network layer implementations (Linear, LayerNorm, RMSNorm, Rotary) reused across model architectures, snapshot-based regression testing for all supported models, and feature-gated builds for precise dependency control. The router provides production features such as CORS, API key authentication, request size limits, health checks, and graceful shutdown. The batching system (`Queue` struct) efficiently accumulates requests and dispatches them to the backend in configurable token-bucket sizes. While the engineering quality is high, the repository exhibits some maintenance risks: significant model duplication across `flash_` and non-flash variants for each architecture, patched upstream dependencies (candle, cudarc) that may diverge from official releases, and commented-out integration tests. The Python backend, wrapping a subprocess, introduces platform-dependent fragility, and the large number of hardware-specific Dockerfiles increases build complexity. Overall, TEI is a well-architected, actively maintained inference server suitable for production deployments, with reusable patterns in its backend abstraction, batch processing, and layer implementations.
FlagAttention is a library implementing memory-efficient attention operators in the Triton language. It provides optimized implementations of flash attention, piecewise attention, paged attention, and gated delta rule operations, designed for use in large language model training and inference. The library features modular architecture with separate implementations for each attention variant, comprehensive testing, and example usage patterns. It leverages Triton for GPU-accelerated computations while maintaining compatibility with PyTorch's autograd system.