Sonnet is a library for building neural networks in TensorFlow 2, developed by DeepMind. It provides a modular abstraction (snt.Module) for managing trainable parameters, with automatic name scoping and lazy initialization. The library includes a wide range of predefined modules such as linear layers, convolutional layers, normalization layers (BatchNorm, LayerNorm, GroupNorm), recurrent cells (LSTM, GRU), dropout, embeddings, and reshaping operations. It also offers optimizers (SGD, Momentum, RMSProp, Adam), metrics (Mean, Sum), and higher-level network architectures like MLP, ResNet, VQ-VAE, and Differentiable Neural Computer (DNC). Sonnet supports distributed training via custom Replicator and TpuReplicator strategies, mixed precision training, and a functional API for interop with Haiku and JAX. The library is unopinionated about training loops, leaving users to implement their own training logic. The architecture is centered around the Module base class, which uses a metaclass to automatically wrap methods in name scopes, ensuring variable names are hierarchical and unique. Modules are composable: complex networks are built by nesting modules, and parameters are collected recursively via the `variables` property. Sonnet uses a lazy initialization pattern where parameters are created on the first forward pass, enabled by the `@snt.once` decorator. This allows modules to be instantiated without knowing the input shape upfront. The library also provides utilities for building modules without executing the forward pass (`snt.build`), applying modules across batch dimensions (`snt.BatchApply`), and deferring module construction (`snt.Deferred`). Key entry points include the top-level `sonnet/__init__.py` which re-exports the public API. Users typically import `sonnet as snt` and use `snt.Module`, `snt.Linear`, `snt.Conv2D`, etc. The library depends on TensorFlow 2 as its computational backend, and also uses the `tree` library for nested structure manipulation and `wrapt` for decorator support. Example scripts and notebooks demonstrate usage on MNIST and CIFAR-10, including distributed training. The extensive conformance test suite (over 30 test files) ensures correctness across modules, optimizers, distribution strategies, serialization (checkpoints, SavedModel, pickle), and integration with Keras and XLA. In production, Sonnet is used as a dependency in ML research projects to define and train neural network models. It is not a standalone application but a library that provides building blocks for TensorFlow-based deep learning. Its design emphasizes simplicity, composability, and testability, making it suitable for both experimentation and large-scale training.
This repository is a comprehensive collection of annotated PyTorch implementations of deep learning papers, spanning transformers, GANs, reinforcement learning, diffusion models, and more. Each implementation includes side-by-side notes explaining the code and underlying concepts, making it an educational resource for understanding and reproducing paper results. The codebase is organized by paper/model type, with standalone experiment scripts, reusable modules, and configuration-driven training loops. Key strengths include clean abstractions for transformer components, a flexible configs system, and extensive coverage of both foundational and modern architectures. However, the project has not seen commits since early 2024, some dependencies are outdated, and certain experiments require large GPU resources or external data not bundled in the repo. While individual components are well-crafted and reusable, the overall collection is best used as a reference and learning tool rather than a production framework.