$ show youweiliang/evit
This repository implements the EViT (Expediting Vision Transformers via Token Reorganizations) method for accelerating Vision Transformers by dynamically reducing the number of tokens processed in intermediate layers through token discarding or fusion. It provides training, evaluation, and visualization scripts for ImageNet classification, leveraging the timm library for core utilities. The codebase includes dynamic token rate scheduling via cosine annealing, throughput and MAC measurement tools, and knowledge distillation support. While the core EViT model is specialized, the repository contains reusable components such as helper functions for scheduling and efficiency metrics (helpers.py) and a knowledge distillation loss wrapper (losses.py). These components are self-contained and demonstrate non-obvious solutions to common problems in efficient deep learning. However, the codebase suffers from technical drift due to outdated dependencies (e.g., torch 1.9.0, timm 0.4.12) and appears inactive based on the commit history.
The LLM identified 2 files worth highlighting for reuse:
Contains utility functions for adjusting token retention rate via cosine schedule, measuring model throughput and MACs, and computing set complements of indices. These functions are self-contained, demonstrate non-obvious solutions for efficient transformer training, and can be reused in other projects requiring dynamic computation reduction or efficiency evaluation.
Implements a DistillationLoss wrapper that combines a base criterion with knowledge distillation loss. This component is self-contained, depends only on PyTorch, and can be easily integrated into other projects requiring knowledge distillation without being tightly coupled to the EViT-specific model architecture.