$ show CultivateLabs/raif
Raif (Ruby AI Framework) is a Rails engine/gem that provides a comprehensive, opinionated framework for integrating large language models (LLMs) into Rails applications. It abstracts over multiple LLM providers (OpenAI, Anthropic, Google Gemini, AWS Bedrock, xAI, OpenRouter) with a unified interface for chat completions, streaming, tool calling, structured outputs, and batch inference. The system persists every LLM interaction as `ModelCompletion` records with full token/cost accounting, supports conversation management with tool call validation and retry loops, and provides a task system for running parameterized LLM prompts with caching and temperature control. A key differentiator is its built-in evaluation framework (`Raif::Evals`) that enables dataset-driven evals with LLM-as-a-judge, statistical comparison (bootstrap CI95, sign test, Fisher exact), and regression gating for CI/CD pipelines. Architecturally, Raif is a mountable Rails engine with its own models, migrations, controllers, background jobs, and admin UI. It uses Single Table Inheritance extensively for extensibility (tasks, agents, model completion batches, LLM judges). The system includes a declarative model manifest DSL for defining model capabilities, pricing, and lifecycle, a data retention/archiving subsystem with pluggable storage adapters, and a Prompt Studio for A/B testing prompts across models. The eval framework supports concurrent execution via forked worker processes with per-worker databases, live HTML reports, and resumable run logs. External dependencies include Faraday for HTTP, ActiveJob for background processing, and provider-specific SDKs (AWS SDK for Bedrock). Key entry points include `Raif::Llm#chat` for direct LLM calls, `Raif::Task#run` for task execution, `Raif::Conversation` for multi-turn conversations, `Raif::Agents::NativeToolCallingAgent` for ReAct-style agent loops, and the CLI (`raif evals`, `raif evals:compare`, `raif evals:report`) for running evaluations. The admin web interface provides full CRUD for tasks, conversations, agents, model completions, and batch inference jobs. Background jobs handle archive/cull operations, batch polling, and cost event repair. The system is designed for production use with features like advisory locks for concurrency control, transient retry with exponential backoff, and configurable data retention policies. While primarily an application framework, Raif has strong data/research characteristics through its evaluation framework, model manifest with pricing/capability tracking, and smoke-testing infrastructure for live API verification. The eval framework produces detailed metrics (pass rates, scores, token usage, costs, confidence intervals) and supports comparison reports with regression gates, making it suitable for systematic LLM evaluation in CI/CD workflows. The model manifest serves as a machine-readable catalog of supported models with lifecycle management, enabling automated capability detection and cost estimation.
The LLM identified 5 files worth highlighting for reuse:
A reusable retry utility implementing exponential backoff with configurable jitter, max retries, retriable exception classes, and Retry-After header support. The pattern is self-contained (no internal Raif dependencies) and can be extracted into any Ruby project needing resilient HTTP or API calls. It demonstrates non-obvious skill in handling transient failures with proper backoff timing, jitter to avoid thundering herd, and respect for server-specified retry delays.
A statistical utility module providing bootstrap resampling for confidence intervals (CI95), exact binomial sign test for paired comparisons, and Fisher exact test for contingency tables. Designed for comparing LLM eval runs, it is fully self-contained and could be extracted as a standalone statistics gem. Demonstrates rigorous statistical methodology applied to LLM evaluation, with proper handling of edge cases like insufficient evidence and Bonferroni correction for multiple comparisons.
A DSL for constructing JSON schemas programmatically, supporting instance-dependent schemas (schemas that vary based on runtime data), dynamic schemas, and arity-aware schema generation. The builder pattern is isolated and could be reused in any Ruby project needing schema generation for structured LLM outputs or API validation. Demonstrates clean DSL design with method chaining and conditional logic.
A forked worker pool for parallel execution of eval tasks, using pipe-based IPC for result collection and error propagation. It manages worker lifecycle, timeout, and result aggregation with thread-safe output buffering. The pattern is self-contained and could be extracted for any Ruby project needing isolated parallel execution with result collection. Demonstrates careful handling of process forking, database connection isolation, and concurrent output without corruption.
A value object representing a data partition for the archive subsystem, encapsulating partition value, SHA-256 token derivation, storage prefix layout, and purge eligibility logic. It is fully self-contained and demonstrates a clean value object pattern for data lifecycle management. The partition logic (configurable column, fallback, token derivation) is reusable in any system needing time-based or ID-based data partitioning for archival.