$ show SalesforceAIResearch/promptomatix
Promptomatix is a framework for automating and optimizing large language model (LLM) prompts through iterative refinement. It provides both CLI and API interfaces for users to input task descriptions, generate synthetic data, optimize prompts using DSPy or meta-prompt backends, and evaluate results with custom metrics. The system features modular architecture separating language model management, configuration, session tracking, feedback handling, and metrics evaluation. Key workflows include prompt optimization with feedback incorporation, batch processing capabilities, and extensibility for custom metrics and optimization backends. The framework targets researchers and developers seeking to reduce manual prompt engineering effort while improving LLM output consistency and quality.
The LLM identified 3 files worth highlighting for reuse:
Abstracts multiple LLM providers (OpenAI, Anthropic, Cohere) behind a unified interface. Encapsulates provider-specific API calls, error handling, and response parsing. Low coupling with core logic (depends mainly on logger and config), making it easily extractable for any LLM application requiring multi-vendor support. Demonstrates clean dependency inversion and encapsulation of varying API contracts.
Manages evaluation of NLP tasks through a pluggable metrics system. Supports standard metrics (BERT Score, ROUGE, etc.) and custom metric registration via a simple interface. Centralizes metric computation, result aggregation, and score normalization. Minimal internal dependencies (primarily logger and external NLP libraries). Exhibits strong separation of concerns and extensibility, allowing reuse in any NLP evaluation pipeline without modification.
Provides reusable text parsing and cleaning functions (e.g., removing extra whitespace, extracting JSON from LLM outputs, normalizing case). Stateless pure functions with zero external dependencies beyond standard library. High cohesion and complete isolation from domain-specific logic. Represents a well-crafted utility module that could be dropped into any text-processing project with no changes, showcasing attention to edge cases and defensive programming.