$ show holoviz/lumen
Lumen is an open-source agent framework for chatting with data and building interactive dashboards, built on the HoloViz ecosystem (Panel, hvPlot, Bokeh). It provides a dual interface: a YAML-based declarative dashboard specification system for non-AI use cases, and an AI-powered natural language interface that generates SQL queries and Vega-Lite visualizations from user prompts. The architecture is modular, with clear separation between data sources (files, SQL databases, REST APIs, xarray), data pipelines (filters, transforms, SQL rewrites), views (hvPlot, Vega-Lite, DeckGL, tables), and an AI agent layer (Planner, Coordinator, specialized agents for SQL, visualization, validation, and source ingestion). The AI system supports multiple LLM providers (OpenAI, Anthropic, Google, Ollama, etc.), vector stores for RAG (Numpy, DuckDB, ChromaDB), and a tool system for external API integration. The codebase demonstrates high craftsmanship with reusable patterns like spec-driven component creation (MultiTypeComponent), composable SQL transforms, and a consistent agent/tool abstraction. While primarily an application framework, it includes data pipeline components that could be extracted for data engineering use. The project is actively maintained with extensive test coverage, though its large optional dependency surface and rapid feature velocity introduce some maintenance risk.
The LLM identified 4 files worth highlighting for reuse:
MultiTypeComponent is a base class that enables declarative, spec-driven component creation with automatic type registration, validation (required keys, type checks), and serialization (from_spec/to_spec). It is the foundation for all Lumen components (sources, views, pipelines, filters, transforms) and can be extracted as a standalone pattern for any Python project needing a plugin-like component system with YAML/JSON round-tripping.
A hierarchy of composable SQL transform classes (SQLLimit, SQLFilter, SQLPreFilter, SQLSample, SQLSchemaStats, etc.) that rewrite SQL queries using sqlglot with dialect-aware parsing and generation. Each transform implements a single responsibility (e.g., adding a LIMIT clause, injecting WHERE filters, sampling rows) and can be chained. The base class provides robust error handling, quoting, and dialect aliasing. This pattern is reusable for any project needing programmatic SQL manipulation with multi-dialect support.
FunctionTool wraps arbitrary Python functions as LLM-callable tools, automatically generating Pydantic schemas from function signatures and docstrings. It supports parameter sanitization, result accumulation across multiple calls, and integration with the LLM tool loop. This pattern is directly reusable for any LLM application needing to expose custom tools with minimal boilerplate.
An abstract VectorStore base class with a consistent API (add, query, upsert, delete, clear, filter_by) and three implementations: NumpyVectorStore (in-memory), DuckDBVectorStore (persistent with HNSW indexing), and ChromaDBVectorStore. Supports metadata filtering, chunking, and contextual augmentation ('situate'). The abstraction allows swapping backends without changing application logic, making it reusable for any RAG system.