$ show openclaw/shellbench
ClawBench is a rigorous execution-first benchmarking framework for evaluating AI agents (particularly OpenClaw) on multi-step tasks. The system orchestrates agent runs through a gateway or containerized native evaluation, scores performance across four axes (completion, trajectory, behavior, advisory judge), and generates detailed diagnostic reports. Its architecture includes a job queue with HuggingFace Dataset persistence, a Gradio web app for interactive leaderboards, and a comprehensive CLI for running evaluations and analyzing results. The framework supports multiple agent runtimes via a pluggable adapter pattern (OpenClaw, Hermes, Claude Code, Codex) and provides a canonical task schema for agent-agnostic task definitions. Beyond basic scoring, ClawBench applies dynamical-systems and statistical methods to agent behavior: trajectory regime classification (trapped, limit-cycle, diffusive), Kaplan-Meier survival analysis, constraint index computation, and variance decomposition. It includes a configuration-space benchmarking system (v0.5) that fingerprints plugin profiles for cold-start prediction and fANOVA factor analysis. The codebase is well-tested with over 60 test files, uses Docker for reproducibility, and includes CI/CD pipelines for automated evaluation campaigns. However, the system is complex and has some hardcoded paths and environment variables tied to specific deployment contexts (Crabbox, OpenClaw gateway), and its heavy dependency on external services introduces maintenance risk. Overall, it is a production-grade benchmark with sophisticated analysis capabilities.
The LLM identified 3 files worth highlighting for reuse:
Defines the AgentAdapter abstract base class with capability gating (AdapterCapability flags), lifecycle methods (setup, run_phase, verify_state_query, teardown), and an @register_adapter decorator for pluggable agent backends. This pattern enables decoupling benchmark orchestration from agent-specific protocols, making it reusable for any new agent runtime. The adapter registration mechanism is clean, type-safe, and isolated from framework internals.
Implements from_task_definition() to convert benchmark-specific TaskDefinition objects into agent-agnostic CanonicalTask format. This bridges the gap between task specifications (with verifier contracts, state queries, budget specs) and any agent runtime that expects a normalized schema. The conversion is deterministic and lossless, enabling reuse across adapters without duplicating task logic.
Applies formal dynamical-systems analysis to agent trajectories: regime classification (convergent/chaotic/trapped), entropy, drift, step size distribution, non-Markov memory metrics, Kaplan-Meier survival curves, and PCA-based bundle analysis. Includes pairwise sensitivity comparisons (score delta, tool edit distance, JS divergence, Lyapunov proxy). This module encapsulates non-trivial statistical methods in a reusable, parameterized interface suitable for any trajectory-based evaluation framework.