SHAP (SHapley Additive exPlanations) is a mature Python library for interpreting machine learning model predictions using Shapley values from cooperative game theory. It provides a unified API across model types: tree ensembles (XGBoost, LightGBM, CatBoost, sklearn, PySpark), deep neural networks (TensorFlow, PyTorch), linear models, and arbitrary callables. The architecture is modular: explainers (Tree, Deep, Kernel, Partition, Gradient, Linear, etc.) consume models and maskers (tabular, image, text) to produce Explanation objects, which are visualized via matplotlib or interactive JavaScript force plots. The library includes benchmarks for evaluating explanation quality, extensive test suites with serialization baselines, and C++/CUDA extensions for performance. Although primarily a library (application), it has a strong research character: it originated from academic papers, includes notebooks reproducing experiments, and contains a user study on human interpretability. The codebase is well-engineered with reusable patterns (serialization, OpChain for lazy operations, MaskedModel abstraction), but carries some technical drift from legacy API aliases, dependency version sprawl, and a large minified JavaScript bundle in the repository.
nnsight is a Python library for interpreting and intervening in neural network models during execution. It provides a tracing API that allows users to access and modify internal activations, compute gradients, and perform interventions like activation patching and steering. The system uses a deferred execution model with thread-based synchronization to run user intervention code alongside model forward passes. Key features include support for Hugging Face models, vLLM integration for high-performance inference, remote execution via NDIF infrastructure, and persistent model editing. The architecture centers around Envoy wrappers for modules, a tracer/interleaver system for coordinating execution, and a batching mechanism for efficient multi-input processing.
Tangermeme is a PyTorch library for post-training interpretability of sequence-based deep learning models in regulatory genomics. It provides a comprehensive suite of functions for analyzing what a trained model has learned about cis-regulatory elements, including attribution methods (DeepLIFT/SHAP, saturation mutagenesis, PISA), sequence perturbation experiments (marginalization, ablation, variant effect, spacing analysis), motif discovery via seqlet calling (recursive and TF-MoDISco-style), and sequence design (greedy/beam search substitution). The library is designed to be assumption-free, extensible to any alphabet, and works with any PyTorch model that operates on one-hot encoded sequences. Key entry points are the `deep_lift_shap`, `predict`, `ersatz`, `marginalize`, `ablate`, `seqlet`, `annotate`, and `design` subpackages, all exposed through a consistent API that accepts a `func=` argument for composability. The library includes a built-in Claude Code skill installer for AI-assisted development, extensive unit tests with toy models, and documentation with tutorials and vignettes. Tangermeme is actively developed (semantic versioning, regular releases) and targets bioinformaticians working with models like BPNet, Enformer, Beluga, or ChromBPNet.