$ show preset-io/promptimize
Promptimize is a prompt engineering evaluation and testing toolkit implemented as a Python library with CLI interface. It enables users to define prompt cases (test cases for LLM prompts) as code, associate evaluation functions that score responses on a 0-1 scale, execute test suites across different models and configurations, and generate performance reports. The architecture centers around BasePromptCase and its derivatives (PromptCase, TemplatedPromptCase, LangchainPromptCase) which encapsulate prompt logic and evaluation, Suite classes that manage collections of prompt cases and handle execution/reporting, and utility modules for templating, caching, and result serialization. The toolkit minimizes redundant API calls by caching prompt executions and only re-running when prompts change, supports human review for edge cases, and integrates with Langchain/OpenAI for LLM interactions.
The LLM identified 2 files worth highlighting for reuse:
Contains reusable evaluation functions for scoring LLM responses: percentage_of_words (calculates match percentage), any_word/all_words (binary word presence checks), and custom all/any aggregators. These functions are self-contained, have clear input/output contracts (string -> float 0-1), and solve a universal prompt evaluation problem. Zero internal dependencies make them trivially extractable for use in any text evaluation system.
Provides several reusable utilities: short_hash/int_hash for generating short unique identifiers, to_yaml/to_json for formatted output with syntax highlighting via Pygments, MeasureDuration context manager for timing operations, and insert_in_dict for ordered dictionary manipulation. These demonstrate solid Python engineering practices, are loosely coupled to the rest of the codebase, and address common programming needs beyond the prompt engineering domain.