Promptimize is a prompt engineering evaluation and testing toolkit implemented as a Python library with CLI interface. It enables users to define prompt cases (test cases for LLM prompts) as code, associate evaluation functions that score responses on a 0-1 scale, execute test suites across different models and configurations, and generate performance reports. The architecture centers around BasePromptCase and its derivatives (PromptCase, TemplatedPromptCase, LangchainPromptCase) which encapsulate prompt logic and evaluation, Suite classes that manage collections of prompt cases and handle execution/reporting, and utility modules for templating, caching, and result serialization. The toolkit minimizes redundant API calls by caching prompt executions and only re-running when prompts change, supports human review for edge cases, and integrates with Langchain/OpenAI for LLM interactions.