$ show openai/simple-evals
This repository is a lightweight, reference implementation of several standard LLM benchmarks (MMLU, GPQA, MATH, HumanEval, DROP, MGSM, SimpleQA, BrowseComp, HealthBench) developed by OpenAI. It provides a unified evaluation framework based on an `Eval` base class and `SamplerBase` that supports OpenAI Chat Completions API, Anthropic Claude API, OpenAI Responses API, and reasoning models (o1, o3, o4-mini). The main entry point is `simple_evals.py` which accepts command-line arguments to select models and evals, runs them, and outputs HTML reports and JSON metrics. The code is cleanly factored into per-benchmark modules, a shared utility module (`common.py`), and sampler adapters. The framework is designed for reproducibility of OpenAI's published benchmark numbers, with all data loaded from public Azure blob storage URLs. The repository is deprecated as of July 2025 and will not receive updates for new models or benchmarks, though it will continue to host reference implementations for HealthBench, BrowseComp, and SimpleQA.
The LLM identified 4 files worth highlighting for reuse:
Provides a set of reusable utility functions for LLM evaluation: `aggregate_results` to combine SingleEvalResults into EvalResult with configurable statistics; `map_with_progress` for parallel execution with progress bar; `check_equality` for mathematical expression equivalence via a sampler; `format_multichoice_question` for standard multiple-choice prompt; `normalize_response` and `normalize_extracted_answer` for cleaning model outputs; and `make_report`/`message_to_html` for generating HTML reports. These functions are decoupled from specific benchmarks and can be extracted into a separate library.
Defines the core abstractions of the evaluation framework: `SamplerBase` (abstract interface for model sampling), `Eval` (abstract base class for benchmarks), `EvalResult` (aggregated results), and `SingleEvalResult` (per-sample results). The `SamplerResponse` dataclass standardizes the output of a model call. These types are minimal, well-documented, and form a clean contract that any new benchmark or sampler can implement.
Implements `SamplerBase` using OpenAI's Responses API, supporting both standard chat models and reasoning models (o3, o4-mini) with configurable reasoning effort. Handles image inputs via `input_image` types, system messages via `developer` role, and retries with exponential backoff on rate limits. This is a practical, self-contained adapter that demonstrates proper usage of the modern OpenAI API and can be reused for any project needing to call OpenAI models with reasoning capabilities.
Implements the HealthBench evaluation, which uses a rubric-based grading system. The `RubricItem` class and `calculate_score` function implement a generalizable pattern for scoring responses against multiple weighted criteria, with both positive and negative points. The `grade_sample` method uses a separate grader model to evaluate each rubric item, aggregates scores per example and per tag (theme, axis), and computes length-adjusted scores. This pattern is reusable for any domain requiring multi-criteria evaluation of model outputs.