$ show mkturkcan/DART
DART (Detect Anything in Real Time) is a training-free framework that converts SAM3 into a real-time multi-class open-vocabulary detector. In practice, the codebase is an inference-optimization and deployment framework: it exports SAM3 backbone components to ONNX/TensorRT, applies FP16/INT8 quantization, builds optimized TRT engines, and runs pipelined video inference with CUDA graphs and torch.compile. It also includes an LLM-driven interactive agent (Qwen via OpenAI-compatible API) that performs iterative segmentation through tool calls (segment_phrase, examine_each_mask, select_masks_and_return), plus video dataset handling, grounding/mask export utilities, and COCO-style evaluation scripts. The repository is a hybrid: fundamentally an application/framework for real-time detection, it carries strong research-code characteristics — calibration caches (calib/*.json), ad-hoc experiment scripts with hardcoded checkpoints (sam3.pt, backbone.onnx), and iterative debugging comments ('Previous test showed...'). The architecture centers on ONNX export of SAM3 blocks (EfficientViT backbone, text encoder, prompt encoder, mask predictor, pixel decoder), TRT engine building, and a quantization layer (dartf/quant/) with experimental plugins tightly coupled to specific ONNX graph patterns. A media production pipeline (dartf/media/) builds explainer videos with automated slide rendering and music mixing. The codebase shows signs of being an experimental workspace rather than a maintained library: hardcoded paths in evaluation configs, version-sensitive TensorRT API usage, a fragile ONNX export pipeline depending on onnxsim subprocess calls, and potential drift between ONNX-exported blocks and real backbone behavior. The 'training-free' claim is partially contradicted by the presence of distillation training code, and the package name dart-detect vs module sam3 mismatch may cause import confusion. Despite these issues, several well-isolated components — an abstract prediction-file evaluator, a TensorRT layer analysis script, and an LLM agent client — could be extracted for reuse in other vision or agent projects.
The LLM identified 4 files worth highlighting for reuse:
OpenAI-compatible LLM client for image analysis. Encapsulates prompt construction, response parsing, and tool-call routing (segment_phrase, examine_each_mask, select_masks_and_return) for iterative segmentation agents. Reusable as a generic LLM-vision client pattern in any project needing agent-driven visual reasoning.
Parses ONNX models into TensorRT network representations and reports layer-type distributions. Self-contained diagnostic utility useful for profiling model composition, verifying ONNX-to-TRT conversion fidelity, and detecting unsupported layers across TRT versions. Extractable as a standalone model-analysis tool.
Abstract base class for prediction-file evaluation, designed to be subclassed for different metrics (e.g., COCO-style, tracking-aware). Demonstrates a clean strategy pattern for pluggable evaluators — a new metric can be added by subclassing without touching evaluation orchestration. The abstraction is generic enough to port to other detection/segmentation evaluation pipelines.
Video dataset loader supporting stage filtering and tracking queries for SAM3-based video segmentation. Handles video-frame sampling, query construction, and tracking-state management. The stage-filtering pattern is reusable for any video-perception pipeline that needs to separate detection, segmentation, and tracking stages.