Docling is a production-grade Python library for converting diverse document formats (PDF, DOCX, PPTX, XLSX, HTML, LaTeX, EPUB, images, audio, video, email, EBCDIC, and more) into a unified, structured `DoclingDocument` representation. It is designed to serve as the document understanding layer for AI workflows, particularly Retrieval-Augmented Generation (RAG), agentic AI, and document extraction pipelines. The system provides a rich Python SDK, a Typer-based CLI (`docling convert`), a service client for remote conversion via `docling-serve`, and an MCP server. It supports multiple output formats including Markdown, JSON, HTML, DocTags, and plain text. Architecturally, Docling employs a modular pipeline system with pluggable backends for each input format and a stage-based processing model. The core pipeline (`StandardPdfPipeline`, `VlmPipeline`, `AsrPipeline`) orchestrates page-level operations (preprocessing, OCR, layout detection, table structure recognition, picture classification, reading order) and document-level enrichments (code/formula extraction, heading hierarchy, picture description). Inference engines are abstracted behind factory patterns supporting ONNX Runtime, HuggingFace Transformers, vLLM, MLX, KServe v2, and remote APIs. The library integrates with over 20 OCR engines and vision-language models, and includes a plugin system for extensibility. Key entry points include the `DocumentConverter` class, the CLI (`docling.cli.main`), and the service client (`docling.service_client`). External dependencies span PDF libraries (pypdfium2, docling-parse), ML frameworks (torch, transformers, onnxruntime), OCR engines (Tesseract, EasyOCR, RapidOCR), and cloud services (KServe, Azure, S3). The codebase is actively maintained with frequent releases, extensive test coverage, and a clear migration path from legacy components.
PaperQA2 is an agentic retrieval-augmented generation (RAG) system for scientific literature. It answers questions by searching, indexing, and analyzing PDFs, text documents, and clinical trials using a multi-step agent workflow. The system integrates with multiple scholarly metadata providers (Crossref, Semantic Scholar, OpenAlex, Unpaywall) for DOI resolution, citation counts, retraction detection, and journal quality scoring. It supports multiple LLM backends via LiteLLM, pluggable PDF readers (pypdf, PyMuPDF, Docling, Nemotron), and vector stores (Numpy, Qdrant). The architecture is a Python library with a CLI (`pqa`), featuring a hierarchical Pydantic settings system, an agent framework with tool selection and memory, and a plugin-based document parsing pipeline. While primarily an application, it is research-backed with published benchmarks (LitQA2, LFRQA) and includes config presets for different quality/performance trade-offs.
BabelDOC is a production-grade PDF translation library and CLI tool that translates documents while preserving original layout, fonts, formulas, and styles. Its core architecture revolves around a custom Intermediate Representation (IL) that decouples PDF parsing from rendering, enabling a modular pipeline of parsing, layout analysis, LLM-based translation, typesetting, and PDF generation. The system supports complex PDF features including CJK text, Type3 fonts, tables, and formulas, and offers both monolingual and dual-language output modes. The translation pipeline is driven by an OpenAI-compatible LLM, with sophisticated prompt engineering for context-aware paragraph translation, glossary integration, and formula/style preservation. Layout analysis uses a DocLayout-YOLO ONNX model for detecting text blocks, tables, and figures. The codebase features a dual parser architecture, retaining a modified pdfminer for compatibility while developing a new native parser with improved font handling and security (CMap pickle deserialization hardening). BabelDOC demonstrates strong software engineering with several genuinely reusable components, including its typed IL data model, a priority thread pool executor, cross-platform memory monitoring utilities, and a high-performance glossary matching system using Hyperscan. The project is actively maintained with frequent releases, comprehensive CI/CD, and responsive security patching. While its complexity and dual-parser architecture present maintenance challenges, the overall code quality, architectural decisions, and reusable patterns make it a valuable reference for PDF processing and LLM-integrated document workflows.
PageIndex is a production-ready RAG framework that replaces vector databases with a reasoning-based, hierarchical tree index over documents. Its core innovation is the 'flash' pipeline: a sophisticated, LLM-free PDF structure extractor that uses character-level layout analysis, trie-based keyword matching, and heuristic classification to build a table of contents tree from PDFs in seconds. The system then enables agentic retrieval by feeding this tree to an LLM, which reasons over the hierarchy to find relevant pages without embedding or chunking. The repository provides a full Python SDK (PageIndexClient) with local and cloud modes, CLI tool, integrations with OpenAI Agents SDK, Anthropic SDK, and Claude Agent SDK, plus an MCP bridge for remote tool execution. While primarily an application, it contains a deep research-grade data processing pipeline (flash) with extensive multilingual dictionaries and layout analysis heuristics, giving it a hybrid character. The codebase is well-structured with clear module boundaries, extensive test coverage, and reusable patterns like the tool contract system and trie-based matching. However, it carries medium technical drift risk due to hardcoded thresholds, dependency on private APIs of LLM frameworks, and pdfium version sensitivity.