$ show infiniflow/ragflow
RAGFlow is a production-grade, open-source Retrieval-Augmented Generation (RAG) engine that combines deep document understanding with agentic workflows. The system ingests heterogeneous documents (PDF, DOCX, images, web pages, etc.), parses them using a multi-stage pipeline (layout analysis, OCR, table recognition), chunks the content with various strategies (token, QA, page, etc.), indexes into vector stores (Elasticsearch, Infinity, OceanBase, SereneDB), and provides retrieval-augmented chat with LLM integration. Beyond basic RAG, it features a visual agent canvas for building complex workflows (parallel subgraphs, loops, tool calls), a sandboxed code execution environment, knowledge compilation (wiki, knowledge graphs, timelines), multi-tenant model management, and chat channel integrations (WhatsApp, Feishu, Discord). The architecture is a full-stack application with a Go backend (Gin), Python API (Flask/Quart), React/TypeScript frontend, and NATS message queue for async tasks. It is designed for enterprise deployment with Docker Compose and supports multiple LLM providers and embedding models.
The LLM identified 4 files worth highlighting for reuse:
A generic, fault-tolerant graph execution engine inspired by Pregel. It manages state across nodes, supports channels (LastValue, EphemeralValue, Topic), checkpointing (memory, NATS), and parallel execution with retry and failover. This component is self-contained within the harness/graph package and could be extracted as a standalone library for building stateful, distributed workflows.
Thread-safe state management for agent canvas execution. Uses sync.RWMutex for concurrent access, supports serialization/export, and provides a clean API for reading/writing variables across nodes. The design is decoupled from the specific agent logic and could be reused in any graph-based execution system requiring mutable state.
A sophisticated token-based chunker that handles multiple document types (text, markdown, JSON, HTML) with configurable delimiters, overlap, and token limits. It includes extensive parity testing against a Python reference implementation and handles edge cases like backtick fences, image IDs, and table context. The chunker is parameterized and could be extracted as a reusable text segmentation library.
An abstraction layer for sandboxed code execution with multiple providers (local Docker, Aliyun, E2B, SSH, Tenki, self-managed). It manages sandbox lifecycle, artifact promotion, and result protocol. The provider interface is clean and extensible, making this a reusable pattern for any application needing isolated code execution.