$ show IBM/AssetOpsBench
AssetOpsBench is a unified open framework for building, orchestrating, and evaluating domain-specific AI agents in industrial asset operations and maintenance (Industry 4.0). It provides a suite of MCP (Model Context Protocol) servers that expose domain tools for IoT telemetry, work order lifecycle, failure mode analysis, time-series forecasting/anomaly detection, vibration diagnostics, and utility functions. The framework includes multiple agent runners (plan-execute, Claude, OpenAI, Deep, Stirrup, OpenCode, direct LLM) that consume these tools, and a comprehensive evaluation pipeline with LLM-as-judge and deterministic structured-answer scorers. The system is designed to run reproducible benchmark scenarios with ground truth, producing leaderboard reports for agent comparison. It is accepted at KDD 2026 and powers multiple competitions. Architecturally, the system is a collection of Python packages with 14 CLI entry points defined in pyproject.toml. Each MCP server is a standalone FastMCP application using stdio transport, backed by CouchDB for persistence. Agent runners share a common trajectory model and observability layer (OpenTelemetry with file-based export). The evaluation pipeline is pluggable with a scorer registry. The codebase also includes a significant data/research component: a time-series feature extraction library (130+ extractors), a model catalog with foundation models (TinyTimeMixer, sktime wrappers), synthetic data generation for vibration signals, and showcase notebooks demonstrating end-to-end model lifecycle. The hybrid nature makes it both a software benchmark framework and a research platform for industrial AI.
The LLM identified 5 files worth highlighting for reuse:
Provides `agent_run_span` context manager that wraps agent execution with OpenTelemetry spans and automatically persists the trajectory to a JSON file on exit. The `persist_trajectory` function writes a structured `Trajectory` object with turn records and tool calls. This component is self-contained (depends only on standard library and opentelemetry-api) and can be extracted into any agent framework to add observability with minimal changes.
Implements a DAG-based recipe composition engine for time-series workflows. Recipes are defined as lists of steps with dependencies; the engine resolves the topological order, executes steps (forecast, anomaly detection, transform, ensemble), and passes results via file-pointer references (HuggingGPT-style). Supports ensemble, conformal backtesting, and lineage tracking. The pattern is reusable for any multi-step analytical pipeline and is tested with in-memory stores.
Pure-numpy implementation of ISO 10816 vibration severity classification and bearing fault diagnostics. Computes RMS, peak, crest factor, and compares against standard thresholds to produce human-readable severity levels (good, satisfactory, unsatisfactory, unacceptable). Also includes bearing fault frequency calculations (BPFO, BPFI, BSF, FTF) and rule-based diagnosis. This module has zero dependencies beyond numpy and can be reused in any vibration analysis pipeline.
A deterministic scorer for structured answers (JSON, counts, choices, mode-based). Supports exact match, partial match, range matching, delta-1 tolerance, and CAR (Correctness, Accuracy, Relevance) scoring for multi-part answers. Parses Python literals and JSON from agent responses. This component is self-contained and can be extracted to evaluate any benchmark with structured ground truth, independent of the LLM-as-judge approach.
Wraps MCP tools to persist large results to workspace files with SHA-256 content hashing and cache invalidation on mutation. Provides a `WorkspaceBridge` that intercepts tool calls, writes outputs to disk, and returns file pointers. The caching layer avoids redundant LLM context consumption. This pattern is reusable for any agent system that needs to manage large tool outputs without overflowing context windows.