$ show promptslab/Promptify
Promptify is a Python library for performing NLP tasks using large language models (LLMs) with structured, type-safe outputs. It provides a scikit-learn-like interface where users instantiate task classes (e.g., NER, Classify, QA, Summarize) and call them with plain text. The library uses LiteLLM as a universal backend to support 100+ providers (OpenAI, Anthropic, Ollama, etc.), Pydantic v2 for output schemas, Jinja2 for prompt templates, and tenacity for retries. Architecture follows a clean separation: Task classes (promptify/tasks) build prompts via PromptBuilder, send them via LLMEngine (which wraps LiteLLM), and parse responses via a safe Parser that never uses eval(). The package also includes a built-in evaluation framework (promptify/eval) with metrics for precision, recall, F1, accuracy, exact match, and ROUGE, plus dataset loading utilities. The code is well-structured, typed, and includes comprehensive tests. The primary artifact is the "promptify" Python package, installable via pip. The repository is not a data research project; it is a software library for developers.
The LLM identified 2 files worth highlighting for reuse:
A safe JSON/Literal parser for LLM output that never uses eval(). It attempts json.loads, falls back to ast.literal_eval, and if parsing fails, can complete incomplete JSON strings by appending closing brackets. This is a reusable, isolated component that solves a common problem in LLM applications: brittle output parsing. It can be extracted and used in any project that needs to recover structured data from LLM text.
A thread-safe cost accumulator that tracks total tokens, cost, and call count across LLM invocations. Uses a dataclass with a lock and provides summary/reset methods. It is self-contained, imports only standard library modules, and can be reused in any project needing aggregated cost tracking for API calls.