Pathway is a high-performance live data processing framework built on top of timely dataflow and differential dataflow, providing a Python API for defining streaming and batch data pipelines. It is designed for real-time ETL, event processing, AI/ML pipelines (including RAG), and incremental computation with strong consistency guarantees. The architecture consists of a Rust core engine that handles dataflow execution, scheduling, persistence, and connectors, while the Python layer provides a declarative API for table operations, UDFs, and integrations with external systems like Kafka, databases, vector stores, and cloud storage. Key entry points include the `pathway` CLI, the Python `pw` module, and the Rust engine library. Data flows through connectors (sources/sinks), transformation operators (joins, groupby, windowing, etc.), and optional persistence for fault tolerance. External dependencies include Kafka, S3, PostgreSQL, various vector databases, and LLM services.
This repository is a collection of deployable templates and educational cookbooks for building real-time LLM applications using the Pathway framework. It provides ready-to-deploy RAG (Retrieval-Augmented Generation) solutions with live data synchronization from various sources (files, Google Drive, SharePoint, etc.), built-in vector and hybrid search, and integration with LangGraph for agentic workflows. The templates are production-oriented with Docker support, while the cookbooks demonstrate advanced patterns like Self-RAG with self-reflection and query rewriting. The codebase is well-structured but exhibits some duplication across templates, and the core logic relies heavily on the external Pathway library.