$ show funstory-ai/BabelDOC
BabelDOC is a production-grade PDF translation library and CLI tool that translates documents while preserving original layout, fonts, formulas, and styles. Its core architecture revolves around a custom Intermediate Representation (IL) that decouples PDF parsing from rendering, enabling a modular pipeline of parsing, layout analysis, LLM-based translation, typesetting, and PDF generation. The system supports complex PDF features including CJK text, Type3 fonts, tables, and formulas, and offers both monolingual and dual-language output modes. The translation pipeline is driven by an OpenAI-compatible LLM, with sophisticated prompt engineering for context-aware paragraph translation, glossary integration, and formula/style preservation. Layout analysis uses a DocLayout-YOLO ONNX model for detecting text blocks, tables, and figures. The codebase features a dual parser architecture, retaining a modified pdfminer for compatibility while developing a new native parser with improved font handling and security (CMap pickle deserialization hardening). BabelDOC demonstrates strong software engineering with several genuinely reusable components, including its typed IL data model, a priority thread pool executor, cross-platform memory monitoring utilities, and a high-performance glossary matching system using Hyperscan. The project is actively maintained with frequent releases, comprehensive CI/CD, and responsive security patching. While its complexity and dual-parser architecture present maintenance challenges, the overall code quality, architectural decisions, and reusable patterns make it a valuable reference for PDF processing and LLM-integrated document workflows.
The LLM identified 8 files worth highlighting for reuse:
Defines the core Intermediate Representation (IL) for the entire pipeline using typed dataclasses (Document, Page, PdfCharacter, etc.). This decouples PDF parsing from rendering, making it a highly reusable and self-contained data model for any PDF processing workflow.
Provides `with_pixel_budget` and `with_target_long_edge` functions for controlling rasterization resolution based on a pixel budget. A clean, self-contained utility for optimizing image processing tasks.
Implements a `PriorityThreadPoolExecutor` that extends Python's `ThreadPoolExecutor` with task prioritization. A well-crafted, reusable concurrency pattern for any application needing prioritized background tasks.
A cross-platform memory measurement utility that uses Linux PSS/RSS with fallbacks. Demonstrates non-obvious skill in system resource monitoring and is safely extractable.
Contains self-contained matrix utilities (`multiply_matrices`, `invert_matrix`) for 2D affine transformations. Essential for any graphics or PDF processing library.
A robust, SQLite-backed translation cache with WAL mode, versioning, and probabilistic cleanup. Demonstrates a solid caching pattern for API-driven services.
Uses the Hyperscan library for high-performance multi-term matching to apply glossary substitutions. A non-obvious pattern for efficient text replacement in translation pipelines.
Provides helper functions for Base85 + Zstd compression and decompression. A specific, well-defined compression pattern that is self-contained and reusable.