$ show eugr/llama-benchy
llama-benchy is a CLI tool for benchmarking OpenAI-compatible LLM endpoints, providing detailed performance metrics similar to llama-bench. It measures prompt processing and token generation speeds at various context depths, supports concurrent requests, prefix caching analysis, and multiple output formats. The tool features a modular architecture with separate modules for HTTP client logic, result processing, and progress reporting. It uses asynchronous HTTP requests to efficiently handle concurrent benchmarking and includes sophisticated logic for handling streaming responses, token timing reconstruction, and latency measurement. The tool is actively maintained with recent commits and up-to-date dependencies.
The LLM identified 2 files worth highlighting for reuse:
Handles all HTTP communication with LLM endpoints, including streaming response parsing, latency measurement, and token timing reconstruction. Demonstrates skill in managing asynchronous HTTP streams, handling server-sent events, and reconstructing token timings when server doesn't provide token IDs. The module is relatively self-contained and could be reused in other LLM interaction tools.
Processes raw benchmark results to compute throughput, latency, and other metrics. Includes sophisticated logic for calculating throughput over time, handling concurrent requests, and dealing with different token streaming behaviors (e.g., burst output, missing token IDs). The module is self-contained and implements non-obvious algorithms for metrics like peak throughput in sliding windows and estimating prompt processing time from measured latency.