Pandoc is a universal document converter written in Haskell. It reads input in dozens of formats (Markdown, LaTeX, HTML, docx, EPUB, Org, RST, MediaWiki, etc.) into a common abstract syntax tree (AST), then writes to a wide range of output formats. The system is modular: readers parse source formats, writers produce target formats, and a template system controls output structure. It includes an integrated citation processor (citeproc), a Lua scripting engine for custom readers/writers/filters, and a server mode (HTTP API). The codebase is a monolith library with separate CLI, server, and Lua engine packages, plus a WebAssembly build for browser use. It is heavily tested with golden tests and property tests. The architecture demonstrates strong separation of concerns, with reusable parsing utilities, a well-defined AST, and a flexible extension mechanism via Lua. While the codebase is large and complex, it is actively maintained and exhibits high craftsmanship in its modular design and handling of diverse formats.
PDFMathTranslate is a production-grade CLI and GUI application for translating PDF documents while preserving scientific formatting (formulas, charts, tables, and fonts). It uses an ONNX-based layout detection model (DocLayout-YOLO) to understand document structure, supports OCR for scanned PDFs, and integrates with numerous translation services (Google, DeepL, OpenAI, Ollama, etc.). The system operates as a monolithic package with multiple entry points: a command-line tool, a Gradio web interface, a Flask+ Celery HTTP API, and an MCP server for AI assistant integration. Internally, it follows a pipeline architecture: input PDF → optional OCR → layout analysis → text extraction → translation (with caching) → typesetting → bilingual output PDF. Notable engineering patterns include a pluggable translation backend via a kernel registry, a configurable translator base class with retry and caching, and a singleton configuration manager. The code is well-structured with test coverage and CI/CD pipelines, though the dependency on a Git submodule for the 'precise' kernel introduces minor coupling risk.
Papra is a self-hostable, open-source document management platform that provides a full-stack solution for archiving, organizing, and searching documents. The system consists of a Hono-based API server with SQLite (via Drizzle ORM and libsql/Turso), a SolidJS web client, a React Native mobile app (Expo), a CLI tool, and several standalone packages. It supports multi-tenant organizations, role-based access, document upload with content extraction (OCR via Tesseract, Mistral OCR, Azure Document Intelligence, Docling, and a custom HTTP strategy), full-text search via SQLite FTS5 with a custom query parser, tagging with automatic tagging rules, document sharing with password-protected links, intake emails, webhooks, and subscriptions with Stripe integration. The architecture is modular, using repository and usecase patterns, an in-process event system, and a driver-based approach for storage (filesystem, S3, Azure Blob), email delivery, and KV stores. The codebase is heavily tested with unit, integration, and e2e tests, and includes a comprehensive demo mode.
ArcadeDB is a high-performance, multi-model database system supporting graph, document, key-value, time-series, vector, and full-text data models. It exposes multiple query languages (SQL, Cypher, Gremlin, GraphQL) and wire protocols (HTTP, Bolt, PostgreSQL, MongoDB, gRPC). The architecture is modular: a core engine (storage, indexing, query execution) with pluggable server plugins for protocols, replication (Raft-based HA), and observability. It features advanced indexing (LSM-tree, hash, vector quantization, sparse vectors, full-text BM25), time-series with PromQL and downsampling, and graph analytical views with CSR compression. Python bindings allow embedded use. The codebase demonstrates strong engineering with extensive use of interfaces, design patterns, comprehensive test coverage (including issue-specific regression tests), and automated CI/CD with performance benchmarks. It is actively maintained with low technical debt and a clean architecture.
PandaWiki is a production-ready, self-hosted AI knowledge base and wiki platform. The actual codebase implements document management with versioned releases, RAG-based Q&A and search, AI-assisted document summarization and editing, multi-source content ingestion (URL, RSS, Sitemap, Notion, Feishu, DingTalk, Confluence, EPUB, Wiki.js, Yuque, Siyuan, etc.), a customizable public landing page builder, analytics, comments, contribution review, and multi-channel bot integrations for WeChat, WeCom, DingTalk, Feishu/Lark, and Discord. It is not a data research repository, but it contains substantial document-ingestion and analytics pipelines that feed the RAG system. The architecture is a full-stack monolith-style application split across a Go backend (Echo + GORM + PostgreSQL + Redis + MinIO/S3 + NATS JetStream), a React admin panel (Vite/MUI/Redux), a Next.js public site, and a pnpm workspace of shared UI/icon/theme packages. The Go backend follows a layered design with handlers, use cases, repositories, middleware, and dependency injection via generated Wire code. A standalone MQ consumer process handles async RAG document updates, crawler exports, and cron-driven statistics aggregation. The public app is served separately from the admin panel, with an embeddable widget and an H5 chat interface. In production, documents are imported through crawler tasks or direct upload, converted to HTML/Markdown, chunked, and upserted into a configurable vector/RAG service. User chat requests rewrite the query, retrieve ranked chunks, and stream LLM responses via SSE with citations. Statistics are rolled up hourly into PostgreSQL. Authentication supports local accounts, JWT sessions, GitHub OAuth, LDAP, CAS, and enterprise SSO. Feature availability is gated by commercial license editions, with a private pro submodule adding enterprise functionality.