vit-pytorch is a comprehensive PyTorch library implementing dozens of Vision Transformer (ViT) architectures and extensions, along with self-supervised pretraining methods (MAE, DINO, SimMIM, MP3, LeJEPA) and video transformers (ViViT, NaViT-3D, CCT-3D). It is a research-oriented codebase offering a consistent API across models, all accepting standard hyperparameters like image_size, patch_size, depth, and heads. The library covers a wide spectrum of attention mechanisms (cross-attention, shifted windows, channel attention, sparse gating), tokenization strategies (patch merging, region-based, t2t, SPT), and training utilities (recorders, extractors, distillation wrappers). While primarily a collection of model implementations, it also includes action transformers for robotics (VAT, VAAT) and N-dimensional ViTs, demonstrating both breadth and depth of recent transformer research. The code is highly modular, relying on shared building blocks such as Attention, FeedForward, and Transformer classes, with heavy but well-organized use of einops for tensor manipulation. Some variants are experimental and may lack the polish of the core models, but the overall architecture is clean and serves as a valuable reference for implementing and comparing ViT variations. The repository is actively maintained, with recent commits and a large user base, but its sheer size and variety mean that not every model is exhaustively tested—core components are covered by basic forward-pass tests, while many advanced variants rely on the underlying patterns being sound.
This repository is an unofficial PyTorch implementation of the Titans architecture from the research paper 'Titans: Learning to Memorize at Test Time'. It provides a library of neural network modules including Neural Memory, Memory-as-Context Transformer, and various attention mechanisms. The codebase consists of modular components that can be used as building blocks for sequence modeling, with training scripts demonstrating language modeling on the enwik8 dataset. The implementation focuses on test-time training mechanisms where the model learns to memorize at inference time through a neural memory module that updates its weights based on incoming sequences.
This repository is a curated collection of Jupyter notebooks implementing core components of modern LLM architectures, intended for interview preparation. It covers attention mechanisms (MHA, GQA, MLA, KV-cache), normalization layers (LayerNorm, RMSNorm), functional components (RoPE, SwiGLU, LoRA, BPE, activation functions, cross-entropy, InfoNCE, sampling, quantization, SFT loss, tool calling parsing), and reinforcement learning losses (DPO, GRPO, DAPO, GSPO, KL divergence). Each notebook is a self-contained, minimal implementation of a specific algorithm, typically using PyTorch. The code is clean, concise, and well-commented, reflecting good engineering practices for educational purposes. While the repository is not a production application and lacks data pipelines, model training scripts, or continuous integration, it serves as a high-quality reference for understanding and reproducing the mathematical foundations of transformer-based language models. The hybrid nature lies in its focus on research-oriented ML components, but its structure is a pure code library rather than a data-driven research project.
This repository implements multiple state-of-the-art Visual Question Answering (VQA) models from 2018, including bottom-up top-down attention, Bilinear Attention Networks (BAN), inter-intra modality attention, learning to count, and conditioned graph structures. It is built on Cyanogenoid's vqa-counting framework and provides a clean, modular codebase for training and evaluating VQA models on the VQA v2 dataset. The code includes data preprocessing scripts (feature extraction, vocabulary building), a flexible training pipeline with various loss functions and learning rate schedules, and evaluation scripts that compute per-category accuracy. While the code is well-structured for research purposes and contains reusable components like the counting module and fusion layers, it relies on deprecated PyTorch APIs (e.g., Variable), hardcoded paths, and lacks unit tests or environment definitions, making reproducibility challenging. The repository is primarily a research artifact for comparing attention mechanisms and counting strategies in VQA.