$ show openclaw/crabpot
Crabpot is a compatibility testing and regression detection framework for the OpenClaw AI agent plugin ecosystem. It validates plugin fixtures (59 plugins, 119 entrypoints) against the OpenClaw API surface by capturing contract registrations, running synthetic probes, measuring cold-import readiness, and profiling resource usage. The system is driven by a modular collection of Node.js scripts (ES modules) that orchestrate inspection, report generation, and CI policy enforcement. It integrates deeply with GitHub Actions for automated checks, track dashboards, and resource campaign execution via Docker. The architecture is script-based rather than a monolithic application, with each script performing a focused task (e.g., capture-contracts.mjs, synthetic-probes.mjs, generate-report.mjs) and producing JSON+Markdown reports. External dependencies include the OpenClaw plugin-inspector library, npm/git for fixture materialization, and GitHub Actions for CI. The framework is designed to catch contract drift before external plugin authors encounter it, and it maintains a dashboard of live issues, compat gaps, and profile regressions.
The LLM identified 5 files worth highlighting for reuse:
Provides robust subprocess execution with process group ownership, timeout escalation, and cleanup for both POSIX and Windows. Uses SharedArrayBuffer and Atomics for coordination, and includes a Windows PowerShell helper. This is a self-contained utility that can be extracted into any Node.js project requiring reliable child process management with guaranteed cleanup.
Core library for building structured reports (JSON and Markdown) from plugin inspector data. Handles evidence linking, submodule references, and deterministic output. The pattern of building a report object and rendering it to both JSON and Markdown is reusable for any CI or audit system that needs human-readable and machine-parseable output.
Generates and executes synthetic probes against captured plugin registrations to verify hook and registrar behavior. Supports failure policies (allowed-blocked, expected-warnings) and produces a probe plan that can be run independently. The pattern of building a probe plan from a contract and executing it with policy enforcement is reusable for any API compatibility testing system.
Defines a contract for resource workloads with paired baseline/enabled cases, phase ordering, and signed delta computation. Validates workload reports against expected structure and derived measurements. This pattern is reusable for any benchmarking system that needs to compare two configurations (e.g., with/without a plugin) and produce signed deltas.
Executes conversation scenarios against an OpenClaw gateway with a mock OpenAI provider. Supports expectation modes (must-pass, known-failure, report-only) and classifies failures by regex patterns. Includes an isolated workspace setup and a WebSocket-based RPC client. This pattern is reusable for any AI agent behavior evaluation that needs to run scripted scenarios with expected outcomes.