Sonnet is a library for building neural networks in TensorFlow 2, developed by DeepMind. It provides a modular abstraction (snt.Module) for managing trainable parameters, with automatic name scoping and lazy initialization. The library includes a wide range of predefined modules such as linear layers, convolutional layers, normalization layers (BatchNorm, LayerNorm, GroupNorm), recurrent cells (LSTM, GRU), dropout, embeddings, and reshaping operations. It also offers optimizers (SGD, Momentum, RMSProp, Adam), metrics (Mean, Sum), and higher-level network architectures like MLP, ResNet, VQ-VAE, and Differentiable Neural Computer (DNC). Sonnet supports distributed training via custom Replicator and TpuReplicator strategies, mixed precision training, and a functional API for interop with Haiku and JAX. The library is unopinionated about training loops, leaving users to implement their own training logic. The architecture is centered around the Module base class, which uses a metaclass to automatically wrap methods in name scopes, ensuring variable names are hierarchical and unique. Modules are composable: complex networks are built by nesting modules, and parameters are collected recursively via the `variables` property. Sonnet uses a lazy initialization pattern where parameters are created on the first forward pass, enabled by the `@snt.once` decorator. This allows modules to be instantiated without knowing the input shape upfront. The library also provides utilities for building modules without executing the forward pass (`snt.build`), applying modules across batch dimensions (`snt.BatchApply`), and deferring module construction (`snt.Deferred`). Key entry points include the top-level `sonnet/__init__.py` which re-exports the public API. Users typically import `sonnet as snt` and use `snt.Module`, `snt.Linear`, `snt.Conv2D`, etc. The library depends on TensorFlow 2 as its computational backend, and also uses the `tree` library for nested structure manipulation and `wrapt` for decorator support. Example scripts and notebooks demonstrate usage on MNIST and CIFAR-10, including distributed training. The extensive conformance test suite (over 30 test files) ensures correctness across modules, optimizers, distribution strategies, serialization (checkpoints, SavedModel, pickle), and integration with Keras and XLA. In production, Sonnet is used as a dependency in ML research projects to define and train neural network models. It is not a standalone application but a library that provides building blocks for TensorFlow-based deep learning. Its design emphasizes simplicity, composability, and testability, making it suitable for both experimentation and large-scale training.
Stable-worldmodel is a research framework for goal-conditioned world models in reinforcement learning, providing a full pipeline from data collection and storage to model training and planning-based evaluation. The codebase is primarily a research platform with strong software engineering practices—pluggable data formats (Lance, HDF5, video, folder), a command-line interface (swm), and a Hydra-based configuration system for reproducible experiments. It includes implementations of multiple world model architectures (LeWM, PLDM, PreJEPA/DINO-WM, TD-MPC2, GCRL), planning solvers (CEM, MPPI, iCEM, gradient-based, Lagrangian), and environment wrappers with domain randomization across 30+ tasks (DeepMind Control, Gymnasium, PushT, OGBench, Craftax, robotics). The repository demonstrates high craftsmanship in its data engineering: the data format registry extends cleanly, the Lance format supports cloud-native streaming from Hugging Face Buckets, and utilities like GoalDataset, HistoryBuffer, and ReplayBuffer solve recurring RL data management problems. The project is actively maintained with extensive test coverage, comprehensive documentation, and a scalable training pipeline using stable-pretraining and Lightning. While the primary goal is research (testing hypotheses about world model generalization under variation), the codebase includes application-level production quality—CI/CD, CLI, checkpoint management, and robust error handling—making it a rare hybrid that serves both researchers and engineers.
Detectron2 is a production-grade, modular computer vision library from Facebook AI Research, designed for object detection, instance/semantic/panoptic segmentation, keypoint detection, and dense pose estimation. It provides a full training and inference pipeline with configurable architectures (Mask R-CNN, RetinaNet, FCOS, etc.), a flexible configuration system (CfgNode and LazyConfig), extensive data loading and augmentation, evaluation metrics for standard benchmarks (COCO, LVIS, Cityscapes), and deployment support via TorchScript and ONNX. The codebase is organized around registries and configurable classes, enabling easy extension. It also includes several research projects (DensePose, PointRend, DeepLab, etc.) as subdirectories, demonstrating its use as a research platform. While the core is well-maintained, some legacy features (Caffe2 export, deprecated LR schedulers) and project-specific code introduce moderate technical drift.
nanochat is a minimal, hackable LLM training and inference framework focused on reproducing GPT-2 performance quickly on 8×H100 GPUs. It provides an end-to-end pipeline from tokenizer training and dataset preparation through pretraining, supervised fine-tuning, reinforcement learning (GRPO/REINFORCE), and a chat CLI. The core is a highly optimized GPT model with advanced features including a custom MuonAdamW hybrid optimizer, FP8 training with tensorwise dynamic scaling, unified Flash Attention 3/SDPA interface, sliding window attention, value embeddings, and BOS-aligned best-fit packing dataloader. Hyperparameters are automatically derived from scaling laws based on model depth, enabling compute-optimal training. The codebase is research-oriented, with extensive experiment logging, leaderboard tracking, and scaling law analysis notebooks. While primarily an application (CLI-driven training/evaluation framework), it has strong data_research elements including dataset pipelines, evaluation tasks (CORE, MMLU, GSM8K, etc.), and analytical notebooks.
ESPnet is an end-to-end speech processing toolkit built on PyTorch, covering a wide range of tasks: automatic speech recognition (ASR), text-to-speech (TTS), speech translation (ST), speech enhancement (SE), speaker recognition, spoken language understanding (SLU), singing voice synthesis (SVS), speech language modeling (SpeechLM), and more. The repository contains both a core Python library (`espnet2/`, `espnet3/`) and a large collection of reproducible recipes (`egs2/`, `egs3/`) for dozens of public corpora and research challenges. The architecture is modular: each task defines its own model, loss, training, and inference via a `Task` class, with abstract base classes for encoders, decoders, frontends, tokenizers, and losses. Configuration is YAML-driven, and the training pipeline uses PyTorch Lightning (in ESPnet3) or a custom trainer (ESPnet2). The system provides CLI tools (`espnet` with subcommands like `transcribe`, `synthesize`, `enhance`, `translate`) and a Python API (`Speech2Text.from_pretrained()`) for inference. It also includes Docker images for deployment and Hugging Face Spaces demos. The repository is hybrid: it is primarily a software framework/library (application) with a strong research character, as evidenced by the extensive recipe collection, benchmark results, and support for state-of-the-art models (Conformer, HuBERT, Whisper, VALL-E). The recipe system follows Kaldi-style data preparation and stage-based pipelines, enabling reproducible experimentation. External dependencies include PyTorch, torchaudio, sentencepiece, and optional components like flash-attn, fairseq, and s3prl. The codebase is actively maintained with CI across multiple OS and Python versions, unit tests, and integration tests.
This repository is the official PyTorch library for Flow Matching generative models, developed by Meta FAIR. It provides a modular and extensible framework for implementing continuous, discrete, and Riemannian flow matching algorithms. The core library consists of four main components: probability paths (defining how data transforms from noise to samples), schedulers (noise/time schedules), solvers (ODE or CTMC integration for sampling), and loss functions (e.g., mixture path generalized KL divergence). It also includes utilities for common operations like model wrapping, manifold geometry (Euclidean, sphere, flat torus), and categorical sampling. The library is designed to be flexible, allowing researchers to easily combine different paths, schedulers, and solvers. Beyond the core library, the repository contains extensive examples for image generation (pixel-space continuous and discrete flow matching on CIFAR-10 and ImageNet) and text generation (discrete flow matching on FineWeb-Edu and Wikitext-103). These examples include full training pipelines, evaluation scripts, and model architectures (UNet, DiT-like transformer). The code is research-grade but well-structured, with clear separation of concerns and thorough unit tests for core components. While the examples are somewhat tied to specific datasets and hardware configurations, the library itself is a reusable foundation for any flow matching project. The repository is a hybrid of a research codebase and a production-ready library. The core modules are abstracted and documented, making them suitable for integration into other projects. However, the examples contain research artifacts like hardcoded paths and experimental features (torch.compile, mixed precision) that may require adaptation. Overall, this is a high-quality, actively maintained resource for anyone working on generative modeling with flow matching.
Open-OmniVCus is a research monorepo reproducing the NeurIPS 2025 OmniVCus method for subject-driven video customization under multimodal control conditions (reference image, depth, mask, motion/camera). It is hybrid by nature: the dominant character is an application/framework layer provided by a large vendored/forked copy of DiffSynth-Studio, wrapped around research-oriented data-construction and training code (VideoCus-Factory, Video-Depth-Anything). The repo does not expose a hosted service or API; it is a Python library plus executable training/inference scripts for diffusion-based image and video generation.
AgentFlow is a framework for building and training agentic LLM systems using reinforcement learning, specifically Flow-GRPO (Flow-based Group Refined Policy Optimization). It provides a modular architecture with four core roles—Planner, Executor, Verifier, and Generator—that coordinate via evolving memory and tool calls over multiple turns. The system supports both inference (via a Solver) and distributed RL training (via a Trainer/Client/Server architecture built on verl, vLLM, and Ray). It includes a multi-provider LLM engine abstraction (OpenAI, Anthropic, Gemini, DeepSeek, etc.), a tool system with metadata-driven discovery, and instrumentation for tracing and reward collection. The repository is heavily research-oriented, with benchmark datasets (AIME24, GPQA, GAIA, HotPotQA, etc.) and training pipelines for optimizing agent planners. While primarily an application framework, it has strong data/research components, making it a hybrid project with a focus on reproducible agentic reasoning experiments.
Visual Spatial Tuning (VST) is a research framework for enhancing Vision-Language Models (VLMs) with human-like visuospatial abilities, from perception to reasoning. It implements a two-stage pipeline: supervised fine-tuning (SFT) on 4.1M spatial perception samples (VST-P) followed by reinforcement learning (RL) on 135K reasoning samples (VST-R). The framework extends to Vision-Language-Action (VLA) models for robotic manipulation evaluated on the LIBERO benchmark. The repository contains custom model implementations (Qwen2.5VL, Qwen3VL) with sequence parallelism and FSDP support, data preprocessing scripts, evaluation benchmarks (VSIBench, 3D object detection on SUN RGB-D, VLM benchmarks), and a distributed RL training system (verl) with reward functions and advantage estimation. While primarily a research codebase, it includes production-quality training infrastructure and reusable data utilities. The code is actively maintained with recent commits, but has medium technical drift due to tight version pinning and reliance on an internal ByteDance framework (VeOmni) that is not publicly available, which may hinder full reproducibility.
Amphion is a comprehensive open-source research toolkit for audio, music, and speech generation, developed by OpenMMLab. It covers multiple generative tasks including text-to-speech (TTS), singing voice conversion (SVC), voice conversion (VC), text-to-audio (TTA), neural audio codecs, and vocoders. The codebase is organized around config-driven experiments (JSON + shell scripts) with modular base classes for trainers, datasets, and inference. It ingests a wide variety of public datasets (LJSpeech, LibriTTS, VCTK, M4Singer, Emilia, etc.) and provides extensive preprocessing pipelines for feature extraction (mel-spectrograms, F0, energy, phonetic alignments, self-supervised features from HuBERT/Whisper). The primary research focus is on high-quality, zero-shot, and controllable generation using architectures like diffusion models, flow matching, GANs, autoregressive transformers, and vector-quantized codecs. The repository also includes evaluation metrics (F0, energy, intelligibility, speaker similarity) and visualization tools (SingVisio). While primarily a research codebase, it includes inference scripts and Gradio demos for practical use. External dependencies include PyTorch, HuggingFace Transformers, WeNet, Descript Audio Codec, and various pretrained models.
OpenHGNN is a comprehensive research framework for Heterogeneous Graph Neural Networks (HGNNs) built on PyTorch and DGL. The repository implements 83+ graph neural network models across 17+ tasks including node classification, link prediction, recommendation, knowledge graph completion, and temporal graph learning. It serves as a benchmarking platform for academic research, providing standardized implementations of state-of-the-art architectures like HGT, MAGNN, CompGCN, and HERO. The framework uses a registration pattern for dynamic model/trainer discovery and supports distributed training, hyperparameter optimization via Optuna, and GraphBolt integration for large-scale graphs. While it has CLI entry points and setup.py, its primary purpose is research reproducibility rather than production deployment. The codebase demonstrates strong architectural patterns with BaseDataset, BaseTask, and BaseFlow abstractions that enforce consistent interfaces across 50+ model implementations. However, significant technical debt exists including version mismatches (setup.py declares 0.7.0 while README claims 0.9.0), hardcoded dataset paths, mixed DGL/PyG usage, and monolithic configuration files. The repository contains extensive training artifacts in output/ directories and documentation drift between README, setup.py, and Sphinx docs. Despite these issues, it remains a valuable resource for graph learning researchers seeking reproducible implementations of heterogeneous GNN architectures.
This repository implements Geometry Forcing, a research framework for video generation with geometric consistency. The core contribution marries video diffusion models (DFoT - Diffusion Forcing Transformer) with 3D representation learning (VGGT) to produce temporally consistent videos conditioned on camera poses. It includes full training pipelines (Hydra + PyTorch Lightning), a suite of video metrics (FVD, FID, LPIPS, VBench), and evaluation scripts for reprojection/revisit error using DROID-SLAM. The codebase is structured as a research template with reusable base classes for algorithms, data modules, experiments, and metric registries. Raw datasets (RealEstate10K, Kinetics-600, Minecraft) are downloaded and preprocessed into video clips with associated camera parameters. Transformations include VAE encoding for latent diffusion, augmentation (flip, reverse), and sliding-window clip extraction. The primary research question is whether video diffusion can be constrained by explicit 3D geometry to reduce inconsistencies in generated videos, measured by geometry-aware metrics. The repo also demonstrates composable sampling via History Guidance and an alignment loss that projects diffusion latents into VGGT feature space. While the code is well-engineered for research (modular design, config-driven, metric registry), it carries typical research code baggage: hardcoded paths, vendored third-party code (DROID-SLAM, VGGT), potential dependency pinning conflicts, and some dead/commented code. Reproducibility is moderate: dependencies are pinned but no environment definitions beyond requirements.txt, and dataset downloads rely on fragile YouTube scraping.