hyperfine is a command-line benchmarking tool designed to measure the performance of arbitrary commands with statistical rigor. It executes commands multiple times, collects metrics such as wall-clock time, CPU time, memory usage, and hardware counters, and provides comparative analysis with confidence intervals and outlier detection. The tool supports warmup runs, parameterized benchmarks (e.g., varying thread counts), and various export formats (CSV, JSON, Markdown) for further analysis. The architecture is modular, separating concerns such as command-line parsing, command representation, benchmark execution, metric collection, and result exporting. It uses a pluggable executor strategy to handle direct command execution, shell-mediated execution (with overhead calibration), and mock execution for testing. Cross-platform support is achieved through conditional compilation and abstractions for platform-specific timing and hardware counter collection. Notable reusable components include utilities for environment randomization to mitigate memory layout effects, a numeric type union for handling integer and decimal parameters, shell syntax detection for safe command parsing, quantity type aliases with serialization support, and cross-platform exit code extraction. The codebase demonstrates strong engineering practices in handling real-world benchmarking complexities like shell overhead, caching effects, and statistical noise.
TOON (Token-Oriented Object Notation) is a human-readable serialization format designed to minimize LLM token usage while preserving lossless round-trip conversion with JSON. This monorepo provides the reference TypeScript implementation (`@toon-format/toon`) and a CLI tool (`@toon-format/cli`) for encoding/decoding TOON. The library features streaming encode/decode, tabular forms for uniform data, normalization of special JavaScript values (BigInt, Date, Set, Map), security hardening against prototype pollution, and a replacer function with path tracking. The CLI supports stdin/stdout conversion, auto-detection by file extension, token statistics, and delimiter options. The repository also includes an extensive benchmark suite comparing TOON against JSON, YAML, XML, and CSV across token efficiency and retrieval accuracy using multiple LLMs (Claude, Gemini, GPT, Grok). Results show TOON matches JSON's accuracy while using ~43% fewer tokens. A documentation site with an interactive playground is provided via VitePress. The codebase is well-structured with clean abstractions, streaming APIs, and thorough testing, though it relies on an external spec repository and has a single author, creating bus factor risk.
clpeak is a cross-platform synthetic micro-benchmark that measures peak achievable compute performance (FLOPS/TOPS), bandwidth, latency, and AI inference throughput across CPUs, GPUs, and NPUs. It supports eight hardware backends: OpenCL, Vulkan, CUDA, ROCm/HIP, Metal, oneAPI/SYCL, native CPU (with per-ISA dispatch), and ONNX Runtime. The tool is architected as a set of backend-specific static libraries orchestrated by a common framework, with two front-ends: a CLI tool (`clpeak`) and a Flutter desktop/mobile GUI. The GUI communicates with the native library via a C-ABI FFI bridge, streaming structured events for live display. Results are serialized to a versioned JSON schema (v3) and can be compared against baselines. The codebase is notable for its meticulous anti-optimization engineering (MAD chains, compiler barriers, calibration probes) and its protobuf-free ONNX model generator. It is a mature, actively maintained project with extensive CI coverage and packaging for multiple platforms.
Vegeta is a versatile HTTP load testing tool and Go library. It simulates constant-rate or oscillating traffic against HTTP endpoints, collects latency and byte metrics, and generates reports (text, JSON, histograms, HTML plots). The core is an attack engine with pluggable pacers, a targeter for reading HTTP request specs, and a metrics pipeline using t-digest for percentile estimates. It includes a Prometheus exporter and a downsampling library for plotting large time series. The codebase is well-structured with clean package boundaries, cross-platform support, and extensive testing including fuzzing.
IMDLBenCo is a comprehensive framework for image manipulation detection and localization (IMLD) research. It provides a unified interface for training, evaluating, and benchmarking deep learning models on standard forensic datasets. The architecture centers around a modular design with registries for models, datasets, and post-processing functions, enabling easy experimentation. Key components include dataset handlers with built-in augmentation and edge mask generation, a distributed training loop with mixed precision support, and evaluation metrics compatible with multi-GPU synchronization. The project serves as both a library for researchers and a reproducible benchmark for state-of-the-art IMLD methods.
This repository from Google DeepMind implements a family of point tracking models (TAP-Net, TAPIR, BootsTAP, TAPNext, TAPNext++, TRAJAN) and the associated TAP-Vid and TAPVid-3D benchmarks. It is primarily a research codebase for tracking arbitrary points in video, with both JAX/Haiku and PyTorch implementations. The code includes training pipelines (Jaxline), evaluation scripts, data generation for synthetic (Kubric) and real datasets (DAVIS, Kinetics, ADT, DriveTrack, Panoptic Studio), and interactive Colab demos. While it has application-like components such as live webcam demos and a VOT challenge tracker, the dominant character is research: the code is organized around benchmarking and model experimentation rather than production deployment. The repository is actively maintained with recent commits and provides extensive documentation for reproducing results.
This repository is a comprehensive benchmarking suite for Rust serialization libraries. It compares over 20 serialization frameworks (e.g., bincode, capnp, flatbuffers, rkyv, prost, serde_json) across four synthetic datasets (log, mesh, minecraft_savedata, mk48) that model realistic data shapes. Benchmarks measure serialization/deserialization speed, payload size, compression (zlib, zstd), and zero-copy access. The codebase includes a deterministic data generation framework via the `Generate` trait, a tooling pipeline (bencher, parser, formatter) for running benchmarks and auto-updating the README with results, and an interactive Yew-based web page for visualizing throughput under bandwidth/CPU constraints. While primarily a research/benchmarking project, it also demonstrates high-quality patterns for conditional compilation, feature-gated code generation, and shared schema definitions across CLI and web frontend.
picosnitch is a Linux daemon for per-executable network bandwidth monitoring using eBPF (CO-RE) and fanotify. It tracks which executables make network connections, records bytes sent/received, timestamps, and metadata (SHA256 hashes, optional VirusTotal and GeoIP lookups). The system provides real-time and historical views via multiple interfaces: a curses TUI (both live top and historical browser), a web UI, and a UNIX socket live feed. The daemon uses a multiprocess architecture with privilege-separated subprocesses for monitoring, event processing, database writes, notifications, and external API queries. It supports local SQLite storage and optional remote SQL (MariaDB/MySQL/PostgreSQL) for tamper-evident logging. The repository also includes a comprehensive comparative benchmark suite (bench/) that tests picosnitch against nine other network monitors across 24 scenarios, with a reusable ToolAdapter framework for running controlled experiments.
This repository is a benchmark suite for comparing the performance of different programming languages in solving the 'related posts' problem: given a list of posts with tags, compute the top 5 related posts for each post based on shared tags. The architecture consists of language-specific implementations (over 50 languages) that all follow the same algorithm: build a tag-to-post-index map, then for each post count shared tags via that map, and select the top 5 posts using a custom priority queue. The repository includes Dockerfiles for each language, helper scripts for running benchmarks, and a workflow that automates execution across multiple post counts (5k, 20k, 60k) and updates a README with results. The code demonstrates strong engineering practices: consistent algorithm across implementations, use of language-specific optimizations (e.g., SIMD, custom hash maps, parallelization), and careful attention to performance measurement (excluding I/O).
This is a research codebase for the Geometric Context Transformer (GCT), a streaming monocular 3D reconstruction model accepted at ECCV 2026. The system processes RGB video frames in real-time (targeting ~20 FPS) to predict camera poses, depth maps, and 3D point clouds using a vision transformer with a novel paged KV-cache causal attention mechanism. It includes a full inference pipeline, interactive and offline renderers (based on Viser and Open3D/CUDA voxelization with octree LOD), and a comprehensive benchmark framework supporting multiple datasets (KTH, Oxford Spires, TUM, 7-Scenes, ETH3D, etc.) with trajectory, depth, and point cloud metrics. The codebase is predominantly application-oriented—providing a deployable model and evaluation harness—but incorporates significant data engineering components such as dataset preprocessing (e.g., Oxford Spires fisheye-to-pinhole conversion with CUDA visibility filtering), standardized BSS artifact storage, and reproducible conda-based environment setup. The core model architecture demonstrates advanced engineering practices: two-stream token layouts, sliding-window KV cache eviction with special-token preservation, 3D RoPE, and modular attention backends (FlashInfer, SDPA, custom causal). The benchmark framework is well-architected with pluggable methods and datasets, resumable execution, and static HTML report generation.
Pyversity is a Python library for diversifying search and retrieval results through re-ranking. It implements five diversification strategies (MMR, MSD, DPP, COVER, SSD) via a unified API, with a focus on reducing redundancy in recommendation and search systems. The library is lightweight, depending only on NumPy, and includes a comprehensive benchmark suite for evaluating strategies on real-world datasets. Architecture-wise, it features a clean separation between the core API, strategy implementations, data types, and utility functions, enabling extensibility and ease of use. The benchmark suite provides a modular pipeline for running experiments, computing metrics (nDCG, MRR, ILAD, ILMD), and generating reports, demonstrating strong engineering practices in experimental design and reproducibility.
AssetOpsBench is a unified open framework for building, orchestrating, and evaluating domain-specific AI agents in industrial asset operations and maintenance (Industry 4.0). It provides a suite of MCP (Model Context Protocol) servers that expose domain tools for IoT telemetry, work order lifecycle, failure mode analysis, time-series forecasting/anomaly detection, vibration diagnostics, and utility functions. The framework includes multiple agent runners (plan-execute, Claude, OpenAI, Deep, Stirrup, OpenCode, direct LLM) that consume these tools, and a comprehensive evaluation pipeline with LLM-as-judge and deterministic structured-answer scorers. The system is designed to run reproducible benchmark scenarios with ground truth, producing leaderboard reports for agent comparison. It is accepted at KDD 2026 and powers multiple competitions. Architecturally, the system is a collection of Python packages with 14 CLI entry points defined in pyproject.toml. Each MCP server is a standalone FastMCP application using stdio transport, backed by CouchDB for persistence. Agent runners share a common trajectory model and observability layer (OpenTelemetry with file-based export). The evaluation pipeline is pluggable with a scorer registry. The codebase also includes a significant data/research component: a time-series feature extraction library (130+ extractors), a model catalog with foundation models (TinyTimeMixer, sktime wrappers), synthetic data generation for vibration signals, and showcase notebooks demonstrating end-to-end model lifecycle. The hybrid nature makes it both a software benchmark framework and a research platform for industrial AI.
SIA (Self-Improving AI) is a framework that orchestrates a three-agent loop (meta-agent, target agent, feedback/improvement agent) to iteratively improve an AI system's performance on a benchmark task. In production, it runs as a CLI tool (`sia run`) that sets up a virtual environment, loads task files, and executes generations: the meta-agent reads `task.md` and generates `target_agent.py`, the target agent executes the task and logs results, and the feedback agent reviews the execution and rewrites the target agent for the next generation. A FastAPI web dashboard auto-launches to visualize per-generation accuracy, agent code, trajectories, and improvement plans. The architecture is a CLI-driven monolith with a plugin system for agent backends (Claude, OpenHands, PydanticAI), JSON-based provider/profile configuration, and Docker sandboxing for secure target-agent execution. It relies on external LLM APIs (Anthropic, OpenAI, Gemini, Nebius, OpenRouter, Together) and optionally on Modal/SandboxFusion for the RL-based weights mode. The repository also bundles four benchmark tasks (GPQA, lawbench, longcot-chess, spaceship-titanic) with public/private data splits and evaluators, giving it a hybrid character that includes data/research components. However, its primary function is as an application for running and visualizing self-improvement experiments.
Debug-gym is a framework for developing and evaluating AI agents that perform interactive debugging of software. It provides a Gymnasium-inspired environment where agents can use tools like a debugger (pdb), file viewer, editor, and shell to inspect and modify code in order to fix bugs. The framework supports multiple execution backends (local, Docker, Kubernetes) to ensure isolation and reproducibility, and integrates with various LLM providers through a unified interface. The system is designed around a loop of observation, action, and feedback: the agent observes the current state (e.g., code, test output), selects a tool to apply (e.g., set a breakpoint, edit a file), and receives the outcome from the environment. This enables the creation of agents that can learn to debug or be evaluated on debugging benchmarks such as SWE-Bench and Mini-Nightmare. The framework also includes analysis tools for visualizing agent trajectories and logging experiment results, supporting both development and research use cases. Key architectural features include a plugin-based tool system, abstract terminal and LLM interfaces, and a secure workspace for code manipulation. These components are designed to be modular and extensible, allowing researchers and developers to plug in new tools, environments, or language models with minimal changes to the core loop.
This repository is a research evaluation harness for Visual Place Recognition (VPR) models. It wraps a wide range of state-of-the-art VPR methods behind a common CLI, allowing users to run any supported model on standard VPR datasets (e.g., St Lucia) or a bundled toy dataset. The main entry point is test_dataset.py, which loads database/query image folders, extracts descriptors using the selected model, computes recall@N metrics, and saves logs and optional visualizations. The repo is designed to work with the companion VPR-datasets-downloader to standardize dataset format. Although it is a CLI tool, its primary purpose is research evaluation, so it is classified as data_research.
This repository provides the Bench2Drive benchmark for closed-loop end-to-end autonomous driving, accepted at NeurIPS 2024 Datasets and Benchmarks Track. It includes a large-scale multi-modal dataset (up to 7.3TB) collected in the CARLA simulator, an evaluation suite with metrics like Driving Score and Success Rate, and baseline results for over 10 methods. The codebase is a hybrid of research data pipeline and application: the dataset generation and evaluation scripts form the core data_research component, while the B2DVisualize Flask app, scenario runner, and leaderboard evaluator provide application-like interfaces for interaction and reproducibility. Data lineage: raw simulation data (sensor recordings, ground truth annotations) is collected via data_collect.py, transformed into compressed archives with JSON manifests, and evaluated using the leaderboard pipeline that computes multiple ability metrics. The repository answers research questions about the effectiveness of end-to-end driving policies under diverse urban scenarios.
DNS Speed Test is a client-side web application that measures DNS-over-HTTPS (DoH) server performance from the user's browser. It implements a single-page architecture with vanilla JavaScript, Tailwind CSS for styling, and Chart.js for data visualization. The core functionality involves building DNS queries, sending them to various DoH endpoints via GET or POST (with CORS handling), measuring round-trip times using performance.now(), and aggregating results into statistics (min, median, average, max). The application stores user preferences (custom hostnames and DoH servers) in localStorage and provides real-time feedback through a responsive UI with sorting, charting, and detailed breakdowns. Key strengths include isolated DNS query construction logic, robust error handling for network timeouts, and a well-tested measurement pipeline. However, the codebase lacks modularity as all logic resides in a single script.js file, limiting reusability without extraction.
SimScale is a research codebase for data-driven, non-reactive autonomous vehicle simulation and benchmarking, developed by OpenDriveLab. It provides a comprehensive pipeline for training and evaluating end-to-end driving policies on the nuPlan/OpenScene dataset, with support for synthetic data augmentation (SimScale) and multiple agent architectures including Transfuser, DiffusionDrive, and GTRS-Dense. The repository implements a Plan-Do-Measure (PDM) closed-loop scoring framework, metric caching, and a Hydra-based configuration system for flexible experiment management. While primarily a research platform, it includes production-like training scripts, evaluation pipelines, and submission tools for the NAVSIM v2 challenge. The codebase is heavily coupled to the nuPlan ecosystem and requires large-scale data downloads (>1TB), but offers reusable components such as abstract agent interfaces, feature/target builders, and batch IDM policies. It is actively maintained with recent commits and a growing community of extensions.
TechEmpower Framework Benchmarks is a large-scale, multi-language benchmark suite for comparing the performance of web application frameworks. The repository contains hundreds of independent framework implementations (e.g., Go, Rust, Java, C#, Python, PHP) each implementing the same standard set of HTTP endpoints: JSON serialization, plaintext, single/multiple database queries, updates, fortunes (HTML), and cached queries. The core benchmark toolset (written in Python) orchestrates Docker-based builds, runs load tests via wrk, and verifies response correctness. The codebase is a federation of many small applications rather than a single system, with each framework's code living in its own directory under `frameworks/<language>/<name>/`. The architecture is highly standardized: every framework exposes the same routes, uses the same database schema (World and Fortune tables), and follows common optimization patterns like connection pooling, prepared statements, batch updates, and template caching. While the repository is actively maintained and the toolset is well-structured, the quality of individual framework implementations varies widely—some are expertly tuned for performance, while others are stubs or use outdated libraries. The primary value of this repository is as a reference for comparative performance numbers and as a collection of idiomatic, performance-focused code examples across many languages and frameworks.
This repository implements Composite, a component-based research operating system where kernel policies (scheduling, memory management, synchronization) are implemented as user-level components. It runs on x86-32, x86-64, and ARMv7a architectures, primarily under QEMU. The system provides a minimal kernel with capability-based security, synchronous invocation (sinv) for inter-component communication, and shared-memory channels. It includes a full musl libc port for POSIX compatibility, a DPDK-based network stack, an lwIP adapter for TCP/IP, and a simple VMM with virtio-net and device emulation. The architecture is closer to a microkernel than a traditional monolithic kernel, with components loaded and composed via a Rust-based composer tool that reads TOML specifications. A large suite of microbenchmarks measures IPC, scheduling, and synchronization latency. The codebase is pre-alpha quality, with many FIXMEs and incomplete features, but demonstrates strong craftsmanship in its abstraction layers (assembly stubs, channel library, scheduler framework, perfdata utility).