$ show NVlabs/SpatialClaw
SpatialClaw is a research framework for spatial reasoning agents — LLM/VLM agents that answer spatial questions about images and video by writing and executing Python code. The core is a LangGraph state machine (`SpatialAgentWorkflow`) that plans, executes code in a Jupyter kernel, receives feedback from tool outputs (SAM3 segmentation, Pi3 depth, Depth Anything 3, MapAnything, 3D reconstruction), and reflects/revises. It is the official implementation of a paper (Cho et al., 2026) from NVIDIA + KAIST AI, reporting 59.9% average accuracy across 20 spatial reasoning benchmarks with six VLM backbones (Qwen3.5/3.6, Gemma4, 26B–397B).
The LLM identified 8 files worth highlighting for reuse:
BaseBenchmark and BaseBenchmarkSample abstract classes defining the evaluation protocol: evaluate_single, evaluate, write_results_summary, get_prediction. Subclasses implement benchmark-specific loading, answer extraction, and scoring. The abstraction makes adding new benchmarks a matter of implementing a few methods.
LLMClient with vLLM auto-discovery: periodic serve.json re-discovery (60s TTL), endpoint pooling with least-connections routing and sticky hash, health tracking with cooldown. Handles model warmup, chat template application, and structured output validation. The pooling/discovery pattern is reusable for any LLM-serving backend.
JupyterKernelManager: persistent IPython kernel with timeout enforcement, namespace reset, variable introspection, and sentinel checking. Manages kernel lifecycle for the agent loop, including interrupt/restart on timeout. The pattern of wrapping a kernel with timeout + state inspection is reusable for any code-execution agent.
AgentTool and CPUTool base classes with GPU inference lock (FIFO mutex), get_prompt_description with sub-section ablation, and serialization support. Tools declare their own prompts, making them self-contained and testable. The locking pattern prevents GPU OOM when multiple tools share a device.
BenchmarkFactory with BENCHMARK_REGISTRY: benchmarks self-register via decorator, and the factory instantiates them by name with config overrides. Adding a new benchmark requires one decorator. The registry pattern is a clean example of plugin-style extensibility.
SLURM chain manager for continuous agent evaluation: submits sequential jobs with overlap, auto-restart on failure (3-consecutive-failure abort), and state persistence. Handles job dependencies, SLURM availability checks, and resume. The chain pattern is reusable for any long-running SLURM workload.
GPU dashboard with Jinja2 templates and Chart.js: overview/history/agent pages with auto-refresh (5s), SLURM job tracking, GPU utilization/memory/power line charts. The dashboard is a self-contained monitoring app that could be extracted for any SLURM GPU cluster.
Converts ReAct tool calls to Python code strings with AST-level safety validation. Enables a structured tool-call executor variant where the LLM emits JSON tool calls that are translated to safe Python. The AST validation pattern is reusable for any LLM code-generation safety layer.