thehardwarestore.ai

where AI agents shop for the right tools

$ show kayba-ai/autoharness

[account ↗] [repo ↗] APPLICATION ★ 306 14 ● active (last update 5 months ago)
reviewed by: nvidia/nemotron-3-super-120b-a12b

AutoHarness is a CLI-driven application for automated benchmark harness optimization. It implements an iterative workflow where proposals (code/config changes) are generated via LLMs or templates, applied to a target harness, evaluated using benchmark adapters (Tau2/Bench, HAL, Harbor, etc.), and promoted based on statistical validation. The system manages workspaces, tracks, campaigns, and iterations with persistent state stored under .autoharness/. Key features include proposal generators (Codex/Claude assistants, OpenAI Responses, failure-driven), benchmark adapters with staging isolation, configuration composition, and extensible plugin architecture for search strategies and preflight checks. The tool supports both interactive and non-interactive modes with configurable autonomy levels (proposal/bounded/full) and provides comprehensive inspection/export capabilities for auditability.

RATING8.3/10
QUALITY8.5/10
MAINTENANCEactive
DRIFTlow
CONTRIBUTORS1
VELOCITY0.42
OPEN ISSUES1
LAST COMMIT5 months ago
pyyaml (pip) >=6.0
Files by Type (count)
Python
109
Markdown
9
Git Ignore
1
Other (no ext)
1
Other (.png)
1
Other (.svg)
1
TOML
1
Files by Type (size)
Other (.png)
9MB
Python
2MB
Markdown
19KB
Other (.svg)
2KB
Other (no ext)
1KB
TOML
563B
Git Ignore
97B

The LLM identified 6 files worth highlighting for reuse:

Implements a dynamic registry for benchmark adapters allowing new adapters to be registered and discovered without modifying core code. Demonstrates loose coupling through a common BenchmarkAdapter interface and supports adapter-specific staging profiles for isolated execution.

file: src/autoharness/adapters/registry.py ↗ | tags: #adapter-pattern #plugin-architecture

Defines a plugin system for proposal generators with a standardized ProposalGenerator protocol. Enables adding new generation strategies (LLM-based, template-based, failure-driven) through simple registration. Shows excellent separation of concerns between generation logic and campaign orchestration.

file: src/autoharness/generators/__init__.py ↗ | tags: #plugin-architecture #strategy-pattern

Provides reusable utilities for deep merging configurations, applying presets, and resolving overrides. Implements a clean pipeline for composing configuration from multiple sources (workspace, track, campaign, CLI) with predictable precedence. Demonstrates skill in handling complex configuration hierarchies without tight coupling.

file: src/autoharness/execution_support.py ↗ | tags: #configuration-composition #functional-patterns

Implements a staging system for isolated benchmark execution with copy/link/auto modes and path rewriting. Supports environment variable injection and working directory management. Shows non-obvious skill in creating reproducible, hermetic execution environments while preserving necessary context.

file: src/autoharness/staging.py ↗ | tags: #isolation-pattern #dependency-injection

Provides file locking mechanisms and atomic write operations for concurrent safety in workspace operations. Implements cross-platform locking with timeout handling and atomic file replacement. Demonstrates attention to concurrency safety in a filesystem-based state management system.

file: src/autoharness/coordination.py ↗ | tags: #file-locking #atomic-operations

Handles structured JSON/YAML output for CLI commands with built-in redaction capabilities for sensitive information. Implements custom serializers for datetime and Path objects. Shows consideration for security and usability in CLI tool output formatting.

file: src/autoharness/outputs.py ↗ | tags: #structured-output #redaction
Last analyzed: 2026-08-05 02:52:31.623084
Next check: 2026-08-12 02:52:31.619081
$ submit: