$ show kayba-ai/autoharness
AutoHarness is a CLI-driven application for automated benchmark harness optimization. It implements an iterative workflow where proposals (code/config changes) are generated via LLMs or templates, applied to a target harness, evaluated using benchmark adapters (Tau2/Bench, HAL, Harbor, etc.), and promoted based on statistical validation. The system manages workspaces, tracks, campaigns, and iterations with persistent state stored under .autoharness/. Key features include proposal generators (Codex/Claude assistants, OpenAI Responses, failure-driven), benchmark adapters with staging isolation, configuration composition, and extensible plugin architecture for search strategies and preflight checks. The tool supports both interactive and non-interactive modes with configurable autonomy levels (proposal/bounded/full) and provides comprehensive inspection/export capabilities for auditability.
The LLM identified 6 files worth highlighting for reuse:
Implements a dynamic registry for benchmark adapters allowing new adapters to be registered and discovered without modifying core code. Demonstrates loose coupling through a common BenchmarkAdapter interface and supports adapter-specific staging profiles for isolated execution.
Defines a plugin system for proposal generators with a standardized ProposalGenerator protocol. Enables adding new generation strategies (LLM-based, template-based, failure-driven) through simple registration. Shows excellent separation of concerns between generation logic and campaign orchestration.
Provides reusable utilities for deep merging configurations, applying presets, and resolving overrides. Implements a clean pipeline for composing configuration from multiple sources (workspace, track, campaign, CLI) with predictable precedence. Demonstrates skill in handling complex configuration hierarchies without tight coupling.
Implements a staging system for isolated benchmark execution with copy/link/auto modes and path rewriting. Supports environment variable injection and working directory management. Shows non-obvious skill in creating reproducible, hermetic execution environments while preserving necessary context.
Provides file locking mechanisms and atomic write operations for concurrent safety in workspace operations. Implements cross-platform locking with timeout handling and atomic file replacement. Demonstrates attention to concurrency safety in a filesystem-based state management system.
Handles structured JSON/YAML output for CLI commands with built-in redaction capabilities for sensitive information. Implements custom serializers for datetime and Path objects. Shows consideration for security and usability in CLI tool output formatting.