$ show datawhalechina/all-in-rag
This repository is a comprehensive Chinese-language educational tutorial and codebase for Retrieval-Augmented Generation (RAG) systems, created by Datawhale. It systematically covers the full RAG stack from fundamentals (data loading, chunking, embeddings) to advanced techniques (hybrid search, query construction, Text2SQL, reranking, agentic RAG, and Graph RAG). The code is organized by chapter, with runnable Python scripts demonstrating each concept using popular frameworks like LangChain, LlamaIndex, and LangGraph, alongside vector databases (Milvus, FAISS) and graph databases (Neo4j). While primarily an educational resource, the repository contains several well-crafted, reusable components such as a minimal Text2SQL agent framework, a clean LangGraph-based agentic RAG pattern, and a custom ColBERT reranker. The practical application is a Chinese recipe Q&A system ('尝尝咸淡'), which serves as a consistent domain for demonstrating various RAG techniques. The codebase is hybrid in nature, blending structured educational content with functional demo applications. The repository is actively maintained with a large community following. However, users should be aware of technical drift risks including vendored model code, hardcoded API dependencies, and the presence of AI-generated code sections. The core value lies in its structured curriculum and reusable RAG patterns rather than as a production-ready system.
The LLM identified 4 files worth highlighting for reuse:
A self-contained, minimal Text2SQL framework (SimpleKnowledgeBase, SimpleSQLGenerator, SimpleText2SQLAgent) that demonstrates a clean separation of concerns for converting natural language to SQL queries. It is isolated, well-structured, and can be extracted and adapted for other database schemas with minimal modification.
Implements a self-reflective RAG agent using LangGraph. The AgentState TypedDict and the graph-based control flow (query analysis -> retrieval -> critique -> refinement -> generation) provide a clean, reusable template for building complex agentic retrieval systems. The pattern is isolated and demonstrates non-obvious skill in state management.
Provides a clean implementation of a ColBERT reranker by inheriting from LangChain's BaseDocumentCompressor. It demonstrates how to integrate a custom, computationally expensive reranking model into a standard LangChain compression pipeline, making it a highly reusable pattern for improving retrieval quality.
A well-structured model loading factory for the EVA-CLIP vision-language model. It handles model configuration registry, pretrained checkpoint loading, and position embedding resizing. While vendored, the factory pattern itself is clean, isolated, and demonstrates strong engineering for managing complex model architectures.