$ show vanna-ai/vanna
Vanna is an open-source Python framework that generates SQL queries from natural language using retrieval-augmented generation (RAG). It provides a modular agent architecture with a core `Agent` class that orchestrates LLM interactions, tool execution, and rich UI component streaming. The system supports multiple LLM providers (OpenAI, Anthropic, Gemini, Ollama, etc.) and vector store backends (ChromaDB, FAISS, Qdrant, Pinecone, etc.) for storing and retrieving training data (question-SQL pairs, DDL, documentation). It includes a comprehensive evaluation framework for comparing agent variants, a legacy Flask API for backward compatibility, and a modern Lit-based web component frontend that renders rich components (tables, charts, cards, progress bars) via SSE streaming. The architecture is highly extensible with dependency injection for lifecycle hooks, LLM middlewares, error recovery strategies, context enrichers, conversation filters, and observability providers. Key entry points include the `Agent.send_message()` async generator, the `ToolRegistry` for tool management, and server factories for Flask and FastAPI. The system relies on external dependencies for LLM APIs, vector databases, and SQL databases, all configured as optional extras. The codebase demonstrates strong engineering practices with clean abstractions, though it carries significant legacy code from a previous version and some placeholder implementations (e.g., MD5-based embeddings in Qdrant).
The LLM identified 4 files worth highlighting for reuse:
Implements `ToolRegistry`, a central registry for tools with access group validation, argument transformation hooks (for row-level security), and execution dispatch. Tools are registered with metadata (name, description, access groups) and can be executed with permission checks. The `transform_args` pattern allows injecting or rejecting arguments based on user context, enabling fine-grained access control. This is a clean, reusable pattern for any agent system requiring tool management and security.
Provides `EvaluationRunner` for comparing multiple agent variants (e.g., different LLMs) on a dataset of test cases. Supports parallel execution with semaphore control, streaming results, and generates `ComparisonReport` with pass rates, average scores, execution times, and token usage. Includes built-in evaluators: trajectory (tool call order), output (content matching), LLM-as-judge (custom criteria), and efficiency. This is a reusable evaluation harness for any conversational AI system.
Defines the `AgentMemory` abstract base class with methods for saving and searching tool usage patterns and text memories. Includes `save_tool_usage`, `search_similar_usage`, `save_text_memory`, `search_text_memories`, and `get_recent_memories`. Multiple implementations (ChromaDB, FAISS, Qdrant, Pinecone, etc.) provide persistent storage with vector similarity search. The interface is clean and can be reused for any agent that needs to remember past interactions.
Implements `CardComponent`, a rich UI component that serializes to a JSON structure consumable by the frontend. Supports header, body, footer, and actions (buttons). Part of a larger `RichComponent` hierarchy that includes `DataFrameComponent`, `ChartComponent`, `TaskListComponent`, `ProgressBarComponent`, etc. These components are yielded by the agent as async generators and rendered by the Lit-based frontend. The pattern demonstrates a clean separation between backend logic and frontend rendering, reusable for any application needing dynamic UI updates.