nnsight is a Python library for interpreting and intervening in neural network models during execution. It provides a tracing API that allows users to access and modify internal activations, compute gradients, and perform interventions like activation patching and steering. The system uses a deferred execution model with thread-based synchronization to run user intervention code alongside model forward passes. Key features include support for Hugging Face models, vLLM integration for high-performance inference, remote execution via NDIF infrastructure, and persistent model editing. The architecture centers around Envoy wrappers for modules, a tracer/interleaver system for coordinating execution, and a batching mechanism for efficient multi-input processing.
This repository implements and evaluates Function Vectors (FVs) for steering large language models (LLMs) without fine-tuning. The core workflow involves computing task-conditioned mean head activations from in-context learning examples, identifying causally influential attention heads via indirect effect, summing their outputs to form the FV, and applying it during inference to elicit target behaviors (e.g., translation, inflection). The system provides a modular Python library with utilities for model loading, prompt construction, activation extraction, intervention, and evaluation, accompanied by a demo notebook and evaluation scripts for running experiments on various linguistic datasets. While the codebase demonstrates clean separation of concerns and contains reusable components for model intervention and prompt handling, it shows signs of technical drift due to dependency rigidity and architecture-specific hooks, and appears to have stagnated in maintenance with no recent updates.