$ show google-deepmind/tapnet
This repository from Google DeepMind implements a family of point tracking models (TAP-Net, TAPIR, BootsTAP, TAPNext, TAPNext++, TRAJAN) and the associated TAP-Vid and TAPVid-3D benchmarks. It is primarily a research codebase for tracking arbitrary points in video, with both JAX/Haiku and PyTorch implementations. The code includes training pipelines (Jaxline), evaluation scripts, data generation for synthetic (Kubric) and real datasets (DAVIS, Kinetics, ADT, DriveTrack, Panoptic Studio), and interactive Colab demos. While it has application-like components such as live webcam demos and a VOT challenge tracker, the dominant character is research: the code is organized around benchmarking and model experimentation rather than production deployment. The repository is actively maintained with recent commits and provides extensive documentation for reproducing results.
The LLM identified 4 files worth highlighting for reuse:
Implements 3D point tracking metrics (3D-AJ, APD, OA) with multiple depth scaling strategies (median, per_trajectory, local_neighborhood, etc.). This module is self-contained and can be extracted for evaluating any 3D point tracking system, not just TAPVid-3D. It demonstrates careful handling of depth uncertainty and occlusion.
Provides `convert_grid_coordinates()` for converting between different coordinate systems (xy, tyx, grid indices). Used consistently across all models and evaluation scripts. This is a small, dependency-free utility that is highly reusable in any computer vision project requiring coordinate transformations.
Contains functions for painting point tracks on video frames, plotting tracks with matplotlib, and estimating homographies with RANSAC for background motion compensation. The homography estimation and RANSAC functions are self-contained and reusable for any video analysis task requiring camera motion estimation.
Implements a clustering algorithm for segmenting point tracks into independently moving objects. Uses a custom forward model with Gram-Schmidt orthonormalization and recursive splitting/deletion. This is a non-trivial pattern for motion segmentation that can be adapted to other point tracking outputs.