$ show allenai/hidden-networks
This repository contains the research code for the paper 'What's hidden in a randomly weighted neural network?' (arXiv 1911.13299). It implements methods for discovering high-performing subnetworks within randomly initialized neural networks without training the weights, building on the lottery ticket hypothesis. The codebase is organized around PyTorch model definitions (ResNet, custom ConvNets), config-driven experiment management (YAML), and modular trainers for different pruning strategies (default, lottery, sample-based sparsification). The core contribution is a set of custom convolution layers (SubnetConv, SampleSubnetConv, FixedSubnetConv) that learn binary masks over frozen weights via straight-through gradient estimators. The repository supports experiments on CIFAR-10, ImageNet, and MNIST, with extensive configuration files for small-scale and large-scale experiments. While the code is well-structured for research reproducibility, it relies on outdated dependencies (PyTorch 1.3.0, torchvision 0.4.1) and contains hardcoded paths, making it primarily useful as a reference implementation rather than a reusable library.
The LLM identified 3 files worth highlighting for reuse:
Defines custom PyTorch convolution layers that learn binary masks over frozen weights. Includes GetSubnet (straight-through estimator for top-k masking), SubnetConv (standard pruning), SampleSubnetConv (binomial sampling for stochastic masks), and FixedSubnetConv (fixed mask after pruning). These layers are the core algorithmic contribution and can be extracted for any PyTorch pruning project.
Implements a Builder class that constructs convolutional layers and batch normalization layers based on configurable types (e.g., DenseConv, SubnetConv, NonAffineBatchNorm). This decouples model architecture from the specific convolution implementation, making it easy to swap pruning strategies. The builder also handles custom weight initialization (signed constant, kaiming normal, etc.) with optional scaling for pruning rates.
Provides utility functions for manipulating model parameters during pruning experiments: freezing/unfreezing weights and subnet scores, setting prune rates across all layers, and computing L1 regularization on scores. Also includes a LabelSmoothing loss module. These functions are reusable for any PyTorch model pruning or fine-tuning workflow.