vit-pytorch is a comprehensive PyTorch library implementing dozens of Vision Transformer (ViT) architectures and extensions, along with self-supervised pretraining methods (MAE, DINO, SimMIM, MP3, LeJEPA) and video transformers (ViViT, NaViT-3D, CCT-3D). It is a research-oriented codebase offering a consistent API across models, all accepting standard hyperparameters like image_size, patch_size, depth, and heads. The library covers a wide spectrum of attention mechanisms (cross-attention, shifted windows, channel attention, sparse gating), tokenization strategies (patch merging, region-based, t2t, SPT), and training utilities (recorders, extractors, distillation wrappers). While primarily a collection of model implementations, it also includes action transformers for robotics (VAT, VAAT) and N-dimensional ViTs, demonstrating both breadth and depth of recent transformer research. The code is highly modular, relying on shared building blocks such as Attention, FeedForward, and Transformer classes, with heavy but well-organized use of einops for tensor manipulation. Some variants are experimental and may lack the polish of the core models, but the overall architecture is clean and serves as a valuable reference for implementing and comparing ViT variations. The repository is actively maintained, with recent commits and a large user base, but its sheer size and variety mean that not every model is exhaustively tested—core components are covered by basic forward-pass tests, while many advanced variants rely on the underlying patterns being sound.
NoPoSplat is a research application for novel view synthesis and camera pose estimation from unposed image pairs. It predicts 3D Gaussian parameters (means, covariances, harmonics, opacities) in canonical space using a vision transformer backbone (CroCo, ResNet, or DINO) with a DPT head, then renders images via differentiable Gaussian splatting. The system eliminates the need for known camera poses during training/inference by estimating poses implicitly through geometric consistency losses. Built on PyTorch Lightning with Hydra configuration, it supports multi-GPU training and evaluation on datasets like RealEstate10K, ACID, DTU, and ScanNet++. The codebase emphasizes modularity through component registries, composable loss functions, and chainable data preprocessing shims.
SAIL-Recon is a feed-forward Transformer-based system for large-scale Structure-from-Motion (SfM) and visual localization. It processes input images or video to estimate camera poses, depth maps, and generate 3D point clouds without iterative optimization. The architecture combines a Vision Transformer backbone with an alternating attention aggregator for temporal modeling and specialized heads for camera, depth, and track predictions. The system includes both command-line and Gradio-based demo interfaces for end-to-end 3D reconstruction, along with evaluation scripts for standard benchmarks (TUM, 7Scenes, Tank&Temple, etc.). While research-oriented, it is packaged as a usable application with clear entry points and modular components.
This repository implements VTP (Visual Tokenizer Pre-training), a framework for pre-training visual tokenizers using a combination of contrastive, self-supervised, and reconstruction learning objectives. The system provides a unified model architecture that supports CLIP-style image-text contrastive learning, DINOv2-style self-supervised learning, and image reconstruction. It includes training scripts, evaluation tools for zero-shot classification, linear probing, and reconstruction metrics, as well as integration with LightningDiT for generation evaluation. The codebase features a HuggingFace-compatible model implementation, modular transformer components, and various utility functions for image processing and metric calculation.
This repository implements the INP-Former model for anomaly detection in industrial inspection tasks. The system extracts Intrinsic Normal Prototypes (INPs) directly from test images using a DINOv2 vision transformer encoder, followed by an INP extractor and decoder to reconstruct normal regions. Reconstruction errors serve as anomaly scores. The repository provides training scripts for various settings (single-class, multi-class, few-shot, zero-shot, super-multi-class), a Gradio-based zero-shot application for interactive inference, ONNX conversion and inference tools, and a competition submission kit. It achieves state-of-the-art performance on standard benchmarks like MVTec-AD, VisA, and Real-IAD. The architecture is modular, separating the core model, dataset handling, utilities, and training logic, though training scripts show some duplication across different settings.
This repository implements Dinomaly, a reconstruction-based unsupervised anomaly detection method for industrial inspection datasets. It uses vision transformers (primarily DINOv2) as feature extractors with a noisy bottleneck and linear attention mechanism. The code supports both class-separated and unified training across MVTec-AD, VisA, and Real-IAD datasets. Key components include dataset loaders, custom optimizers, training utilities, and evaluation metrics. While the research code achieves state-of-the-art results, it contains hardcoded paths, version-specific dependencies, and dataset-specific logic that limit immediate reproducibility.
This repository implements the EViT (Expediting Vision Transformers via Token Reorganizations) method for accelerating Vision Transformers by dynamically reducing the number of tokens processed in intermediate layers through token discarding or fusion. It provides training, evaluation, and visualization scripts for ImageNet classification, leveraging the timm library for core utilities. The codebase includes dynamic token rate scheduling via cosine annealing, throughput and MAC measurement tools, and knowledge distillation support. While the core EViT model is specialized, the repository contains reusable components such as helper functions for scheduling and efficiency metrics (helpers.py) and a knowledge distillation loss wrapper (losses.py). These components are self-contained and demonstrate non-obvious solutions to common problems in efficient deep learning. However, the codebase suffers from technical drift due to outdated dependencies (e.g., torch 1.9.0, timm 0.4.12) and appears inactive based on the commit history.