OpenDataLoader-PDF is a production-grade, multi-language PDF-to-structured-content extraction tool designed for AI/LLM pipelines, RAG systems, and accessibility workflows. It converts PDFs into JSON, Markdown, HTML, plain text, and tagged PDF (PDF/UA) with high accuracy (0.907 overall benchmark). The core engine is written in Java (veraPDF-based) and provides a rich CLI, while Python and Node.js SDKs bundle the Java JAR and expose idiomatic APIs. A hybrid mode routes pages to an external AI backend (Docling) for OCR, formula extraction, and picture descriptions, with a triage processor deciding per-page routing based on content signals. The project also includes an MCP server for AI agent integration, a LangChain loader, and an agent skill bundle for correct usage.
This is a monorepo of six AI agent skills (SKILL.md format) for creative content generation, targeting Claude Code, Cursor, Codex CLI, and other compatible agents. Each skill is a self-contained package with agent instructions, reference libraries, scaffold scripts, and optional companion websites. The skills enable agents to: generate beautiful single-file HTML articles (beautiful-article), create and edit images via GPT-4o (gpt-image-2), retrieve knowledge from local files (kb-retriever), produce deterministic motion-graphics videos with audio (motion-showreel), design polished web pages with anti-AI-cliché rules (web-design-engineer), and build click-driven video presentations with TTS (web-video-presentation). The repository also includes release automation (zero-dependency Node scripts, GitHub Actions), a Claude Code plugin marketplace entry, and two showcase websites (Vite+React+TypeScript). While primarily an application toolkit, it contains structured prompt template libraries and design reference catalogs that resemble curated data assets. The codebase demonstrates high engineering craftsmanship with deterministic rendering pipelines, a session-based audio synthesizer, design token systems, and comprehensive test coverage for the motion-showreel pipeline.
Neuron AI is a PHP framework for building AI-powered applications with support for multiple large language model providers (OpenAI, Anthropic, Gemini, Mistral, Ollama, AWS Bedrock, etc.), tool integration, structured output, retrieval-augmented generation (RAG), and complex workflow orchestration. It provides a composable agent class that can operate in chat, stream, and structured output modes, a workflow engine with interrupt/resume and persistence, a built-in evaluation framework with LLM-based judges and assertions, and a Model Context Protocol (MCP) client for connecting to external tool servers. The architecture is modular: agents are built on top of workflows, tools are defined via PHP classes with JSON schema generation, and providers are abstracted behind a common interface with traits for shared logic.
Flowcraft is a workflow orchestration framework for building, executing, and monitoring durable workflows. It provides a type-safe, composable API for defining workflows as directed graphs of nodes and edges, supporting both programmatic (fluent) and declarative (JSON) definitions. The framework enables progressive scaling from in-memory execution to distributed systems via adapters for message queues and cloud services, with built-in support for patterns like retries, timeouts, human-in-the-loop, batch processing, and subflows. Key architectural pieces include a dependency-free core runtime, a sophisticated context tracking system for efficient state persistence, a plugin-based adapter architecture for distributed execution, and LLM tooling for workflow introspection and control via natural language.
LlamaIndex is a comprehensive Python framework designed for building Large Language Model (LLM) applications, with a primary focus on Retrieval-Augmented Generation (RAG) and agentic workflows. It functions as a modular library rather than a monolithic application, providing abstractions for data ingestion, indexing, retrieval, and query orchestration. The architecture is split between a core package (`llama-index-core`) containing base interfaces and logic, and a vast ecosystem of integration packages (`llama-index-integrations`) that connect to external LLM providers, vector databases, and data sources. The codebase demonstrates a strong plugin-based design, allowing developers to swap components like embeddings, retrievers, and LLMs without rewriting the core orchestration logic.
Milvus is a high-performance, distributed vector database designed for AI applications such as vector similarity search, RAG pipelines, and hybrid search (dense + sparse). It follows a microservices architecture with components like proxy, rootcoord, datacoord, querycoord, datanode, querynode, indexnode, and streamingnode. The system uses a Write-Ahead Log (WAL) with multiple backends (RocksMQ, Kafka, Pulsar, Woodpecker) for real-time ingestion, and a C++ execution engine (segcore) for vector indexing and query processing. It supports a wide range of index types (IVF, HNSW, DiskANN, SCANN, etc.), scalar indices, full-text search via Tantivy (Rust), and advanced features like CDC replication, RBAC, and multi-tenancy. The codebase is primarily Go and C++, with Python test suites and Rust bindings. It is actively maintained with frequent commits, extensive tests, and a robust client SDK.
DeepTutor is a full-stack AI-powered learning companion that provides chat, problem-solving, research, visualization, and structured learning capabilities. The system features a modular architecture with a Python/FastAPI backend, Next.js/React frontend, and a CLI interface. It implements an agent-based system where capabilities like deep solve, deep question, and deep research are implemented as loop capabilities with access to tools such as web search, code execution, and knowledge base retrieval. The platform includes a three-layer memory system for contextual awareness, supports multiple RAG backends (LlamaIndex, PageIndex, GraphRAG, LightRAG, IMA), and enables integration with external knowledge bases (Obsidian, MarginNote 4) and partner agents (Claude Code, Codex). The application employs a plugin architecture for extensibility, allowing new capabilities, tools, and subagents to be added via entry points. Configuration is managed through JSON/YAML files with runtime overrides, and services are decoupled via provider abstractions for LLMs, embeddings, search, and media generation. Real-time communication is facilitated by WebSockets and an internal event bus (StreamBus). The system supports multi-user deployments with role-based access control, isolated workspaces, and grant-based resource sharing for skills, knowledge bases, and partners. Deployment options include local installation, Docker containers, and CLI-only mode. The codebase demonstrates strong engineering practices with extensive unit testing, clear separation of concerns, and reusable patterns such as the capability protocol, tool composition, and atomic file operations. While the system's feature richness introduces complexity, its modular design allows for selective adoption of components.
Haystack is an open-source Python framework for building production-ready LLM applications. It provides a modular, component-based architecture where data processing steps (converters, embedders, retrievers, generators, evaluators, etc.) are connected into pipelines or agents. The framework includes over 100 built-in components, support for 50+ integrations (OpenAI, Anthropic, Azure, HuggingFace, etc.), async execution, streaming, serialization, evaluation, and observability. It is designed for RAG, semantic search, question answering, autonomous agents, and multimodal applications. The code is well-structured with consistent patterns (component decorator, lifecycle methods, serialization protocols), extensive test coverage, and security hardening (fuzzing, sandboxed Jinja2, deserialization allowlists).
EverOS is a local-first, markdown-based memory framework for AI agents and chat applications. It ingests conversational sessions via a FastAPI REST API, extracts structured memories (episodes, atomic facts, agent cases, skills, user profiles) using LLMs, persists them as version-controllable Markdown files, and maintains derived search indexes in LanceDB (vector + BM25) and SQLite. The system provides a rich retrieval layer with keyword, vector, hybrid (RRF), and agentic (LLM-guided multi-round) search methods, plus a knowledge wiki feature. An Offline Memory Engine (OME) runs background strategies (reflection, clustering, extraction) on cron/idle/immediate triggers, and a cascade daemon watches the Markdown filesystem to reconcile indexes. The architecture is a layered monolith (entrypoints → service → memory → infrastructure) with cross-cutting components for LLM, embedding, rerank, tokenizer, and config. It ships with a CLI (Typer), a TUI demo (Textual), and extensive integration tests. While primarily an application, it includes a benchmark harness (LoCoMo, LongMemEval, EverMemBench) and evaluation fixtures, giving it a hybrid character with a strong research-oriented testing pipeline.
Kiln is a full-stack AI development platform for building, evaluating, and optimizing LLM-based tasks and agents. It provides a desktop application (Tkinter tray + FastAPI server) with a Svelte web UI, plus a core Python library and a cloud server component. The system supports the complete AI workflow: project/task management, multi-provider LLM orchestration via LiteLLM, evaluation with multiple judge types (LLM-as-judge, code eval, exact match, pattern match, etc.), synthetic data generation, RAG pipelines (chunking, embedding, vector stores), fine-tuning, prompt optimization, and Git-based project versioning. It also includes a 'Kiln Pro' cloud service for advanced features like copilot-assisted spec refinement and batch generation. The architecture is a monorepo with uv workspace, featuring extensive test coverage (unit, integration, e2e) and CI/CD pipelines. While the primary character is an application, it incorporates significant data engineering patterns for evaluation pipelines, synthetic data generation, and statistical analysis.
Ask Molty is a Cloudflare Worker that serves as a documentation agent for OpenClaw. It accepts user questions via a chat API, builds a read-only workspace from pre-indexed documentation, source code, and GitHub issues/PRs, then uses OpenAI's chat models with tool calls (search_workspace, read_workspace, list_workspace, run_shell) to answer grounded in the workspace. The worker handles authentication via OpenClaw ID or GitHub session cookies, serves artifact files (e.g., github-search.jsonl) via a proxy, and includes robust error handling for timeouts, malformed data, and streaming. The codebase is well-structured with separate modules for auth, prompt, retrieval, and types, and includes a comprehensive smoke test suite that mocks network interactions to verify behavior.
CherryStudio is a cross-platform desktop AI assistant built with Electron and React. It provides a unified chat interface that supports dozens of LLM providers (OpenAI, Anthropic, Google, Groq, local models via Ollama, etc.) through a pluggable provider registry and the Vercel AI SDK. Beyond chat, it offers knowledge base management with RAG (vector indexing via sqlite-vec, chunking, embedding), file processing (OCR with Tesseract/PaddleOCR, document parsing), web search, image generation, MCP (Model Context Protocol) tool integration, and code CLI management (Claude Code, OpenAI Codex). The application is organized as a pnpm monorepo with separate packages for the UI component library, AI core runtime, provider registry, and tracing infrastructure. Architecturally, the app follows an Electron main/renderer process split with typed IPC communication using Zod schemas. The main process handles AI inference, database operations (SQLite via Drizzle ORM), file management, and native integrations. The renderer is a React SPA using TanStack Router, SWR for data fetching, and a custom UI component library with extensive icon sets for providers and models. The AI runtime layer abstracts provider-specific details through adapters and supports streaming, tool calls, agent sessions, and context compaction. Key data flows include: user input → composer → IPC → AI runtime → provider API → streaming response → renderer. Knowledge base ingestion follows: file upload → OCR/parsing → chunking → embedding → vector store. The application also supports offline usage with local embedding models and ONNX runtime. External dependencies include native binaries for OCR and vector search, and the AI SDK ecosystem.
Vane is a privacy-focused, self-hosted AI answering engine that combines web search with large language models (LLMs) to produce cited, real-time answers. The system is built as a Next.js 16 application with a React frontend and a set of API routes that orchestrate a multi-step research pipeline: classification of user queries, parallel execution of search actions (web, academic, social, scrape, upload-based retrieval), widget generation (weather, stock, calculation), and final answer synthesis via an LLM. It supports multiple LLM providers (OpenAI, Anthropic, Gemini, Groq, Ollama, LM Studio, etc.) through a clean provider abstraction, and stores chat history in a local SQLite database managed by Drizzle ORM. The architecture is a monolith with clear separation of concerns: a model registry for LLM/embedding providers, a search agent with an action registry pattern, a widget system for structured data display, and a configuration manager that persists user settings to a JSON file. The frontend uses client-side streaming via EventSource to display research progress and final answers in real-time. Deployment is Docker-first, bundling a SearXNG instance for privacy-respecting web search, with multi-arch images and a slim variant. Key entry points include the main chat page (`/`), individual chat pages (`/c/[chatId]`), a news discovery page (`/discover`), and a library for uploaded documents. The primary API endpoints are `POST /api/chat` (streaming chat), `POST /api/search` (non-streaming search), and various CRUD routes for providers, config, and uploads. External dependencies include SearXNG (self-hosted), Open-Meteo (weather), Yahoo Finance (stocks), and the LLM provider APIs. The codebase demonstrates strong engineering practices with well-defined abstractions, a modular agent architecture, and a focus on privacy and local-first operation. It is actively maintained with a large community, though some areas (custom migration runner, large chat hook) show room for refinement.
GPT Researcher is an autonomous research agent that produces comprehensive, cited reports from web and local sources. In production, it operates as a full-stack web application: a FastAPI backend (main.py, backend/server/app.py) exposes REST and WebSocket endpoints for initiating research, streaming progress, and retrieving reports. A Next.js frontend (frontend/nextjs/) provides the user interface, and a lightweight vanilla JS frontend (frontend/scripts.js) is also available. The core Python package (gpt_researcher/) can be used programmatically via the GPTResearcher class or as a CLI tool (cli.py). Architecture follows a planner-executor pattern: given a research task, the system generates sub-queries, executes them in parallel across 20+ search engine wrappers (Tavily, Google, Bing, Brave, DuckDuckGo, ArXiv, PubMed, etc.), scrapes the resulting URLs using multiple backends (BeautifulSoup, browser automation, PyMuPDF, FireCrawl), filters and compresses the context using embedding or keyword-based methods, and finally generates a structured report with citations. The system supports multiple report types (summary, detailed, deep research) and multi-agent orchestrations (LangGraph, AG2) for more complex workflows. Key data flows: user query to sub-query generation (LLM) to parallel search across configured retrievers to URL scraping to context compression to source curation to report generation (LLM) to output as markdown/PDF/DOCX. Real-time progress is streamed via WebSocket. External dependencies include LLM providers (OpenAI, Anthropic, Google, Ollama, etc.), search APIs (Tavily, SerpAPI, etc.), and optional services (MCP servers, vector stores, image generation APIs). The codebase demonstrates strong engineering practices: extensive test suite (200+ tests), plugin architecture for retrievers, configurable context filters, cost tracking, and evaluation frameworks for quality and hallucination. However, it carries maintenance risks from its large dependency surface, multiple parallel agent implementations, and reliance on rapidly-changing external APIs.
FastGPT is a production-grade AI Agent building platform that enables users to create, deploy, and manage AI applications through a visual workflow editor. The system provides a full-stack solution including a Next.js frontend with Chakra UI, a comprehensive REST API layer, and a modular backend architecture. Core capabilities include a workflow engine with agent loops, tool calling, dataset (knowledge base) management with vector search, sandboxed code execution (JS/Python), multi-tenant team management with fine-grained permissions, billing and usage tracking, and integration with external platforms (WeChat, Feishu, Dingtalk, MCP). The platform is designed for both open-source community and commercial deployments, with feature gating via license activation. The codebase is organized as a monorepo with clear separation of concerns: shared types and constants in `packages/global`, business logic in `packages/service`, UI components in `packages/web`, data access layer in `packages/dal`, and separate services for sandbox execution and volume management. It relies on MongoDB, Redis, BullMQ, S3-compatible storage, and multiple vector database backends (PG, Milvus, OceanBase, SeekDB, OpenGauss). The architecture is predominantly monolithic with the main application as a Next.js app, but sandbox and volume manager run as independent services, indicating a hybrid monolith/microservices pattern.
PaperQA2 is an agentic retrieval-augmented generation (RAG) system for scientific literature. It answers questions by searching, indexing, and analyzing PDFs, text documents, and clinical trials using a multi-step agent workflow. The system integrates with multiple scholarly metadata providers (Crossref, Semantic Scholar, OpenAlex, Unpaywall) for DOI resolution, citation counts, retraction detection, and journal quality scoring. It supports multiple LLM backends via LiteLLM, pluggable PDF readers (pypdf, PyMuPDF, Docling, Nemotron), and vector stores (Numpy, Qdrant). The architecture is a Python library with a CLI (`pqa`), featuring a hierarchical Pydantic settings system, an agent framework with tool selection and memory, and a plugin-based document parsing pipeline. While primarily an application, it is research-backed with published benchmarks (LitQA2, LFRQA) and includes config presets for different quality/performance trade-offs.
AutoRAG is a production-grade, privacy-focused personal AI assistant and retrieval-augmented generation (RAG) agent. It provides a local-first search and answer engine that indexes federated datasources (Slack, Discord, KakaoTalk, GitHub, email, Obsidian, RSS, cloud drives, and more) via external CLI connectors, without migrating data. The system exposes a CLI, MCP server, TypeScript API, and a TUI (Pi interactive host), and powers a separate desktop Electron app. Its architecture is modular: a configurable agent (`AutoRAGAgent`) orchestrates multiple retrieval methods (BM25, semantic vectors, hybrid RRF, file-name search, web search), document parsing (PDF, DOCX, HWP, images via OCR), embedding (local Qwen3 model or external APIs), reranking (OpenRouter/Dazzi), and evidence judgment. A peer-to-peer layer (SimpleX) enables policy-controlled file sharing with injection and PII gates. The repository also contains a legacy Python codebase (`legacy/`) focused on automated RAG pipeline evaluation and QA dataset generation, which is in maintenance mode but still provides reusable metrics and pipeline components. This is a hybrid project where the dominant character is a modern TypeScript application, with significant data/research heritage in the legacy Python portion.
This repository contains 'officeParser', a robust Node.js and browser library for parsing 12 office document formats (DOCX, PPTX, XLSX, ODT, ODP, ODS, PDF, RTF, CSV, MD, HTML, EPUB) into a unified abstract syntax tree (AST) and generating high-fidelity output in 9 formats (JSON, text, Markdown, HTML, CSV, RTF, PDF, EPUB, and RAG-focused chunks). It provides a first-class CLI (`npx officeparser`) and a programmatic API (`parseOffice()` → AST → `.to(format, config?)`), with extensive configuration for OCR, chunking strategies, security limits, and format-specific dialects. The architecture follows a strict pipeline: format-specific parsers normalize documents into `OfficeParserAST`, then format-agnostic generators produce the desired output, enabling cross-format conversion and round-trip fidelity. The codebase demonstrates high craftsmanship with reusable utility modules (sanitization, typed error/warning systems, lazy ZIP handling, MathML-to-LaTeX conversion) and a comprehensive test suite covering parser baselines, generator baselines, security injection tests, and exhaustively tested edge cases. It supports browser and Node runtimes via dual CJS/ESM packages and bundled builds, including slim variants that minimize dependencies for browser environments. Key data flows include document metadata extraction, attachment handling (images, charts), OCR via Tesseract.js, PDF generation via Puppeteer, and semantic chunking for LLM ingestion. External dependencies include `pdfjs-dist`, `tesseract.js`, `fflate`, `chart.js`, and `puppeteer` (optional). While the repository is actively maintained with recent commits and CI across multiple Node versions, it shows some technical drift: deprecated API fields remain documented alongside replacements, and large files (e.g., OpenOfficeParser.ts at ~1600 lines) increase maintenance complexity. The builder has addressed past packaging issues (webpack context-module) and maintains sophisticated browser shims. Overall, it is a production-grade document conversion toolkit with exceptional reusability in its core utilities and a clean separation between parsing and generation.
JeecgBoot is an enterprise-grade low-code development platform that accelerates application delivery through AI-powered code generation, online form/report builders, and workflow integration. The backend is built on Spring Boot 3 with MyBatis-Plus, Shiro, and JWT, supporting both monolithic and microservice architectures (Spring Cloud, Nacos, Sentinel, XXL-Job). The frontend uses Vue 3 + TypeScript + Ant Design Vue with a rich component library including JVxeTable for editable grids and schema-driven forms. The platform provides comprehensive RBAC, multi-tenancy, dynamic data sources, data dictionaries, and third-party integrations (DingTalk, WeChat, Feishu). A dedicated AI module (airag) offers RAG knowledge bases, LLM chat, voice/image generation, and MCP plugin support. The codebase is actively maintained with frequent commits and a large community, though it contains some commented-out legacy code and hardcoded security values that represent minor technical debt.
Open Notebook is a self-hosted, AI-powered research assistant and knowledge management application. It allows users to collect sources (web pages, uploaded documents, text snippets), extract structured insights via LLM-driven transformations, organize content into notebooks, perform hybrid text/vector search, and generate AI podcasts with multi-speaker support using TTS models. The backend is built on Python with FastAPI and LangGraph for orchestration, SurrealDB for persistence (with a custom migration system), and the Esperanto library for multi-provider AI integration (OpenAI, Anthropic, Google, Ollama, and many others). The frontend is a modern Next.js application with React Query, i18n support for 14 locales, and a component library based on Radix UI/shadcn. The architecture follows a monolith-with-background-workers pattern: the FastAPI process handles both synchronous API requests and, via the surreal-commands library, asynchronous background jobs for content processing, embedding generation, and podcast generation. The codebase is well-structured with clear separation between API routers, domain models, utility modules, and frontend components. It exhibits strong security awareness—includes SSRF protection via DNS pinning, path containment for file uploads, Fernet encryption for stored credentials, and input validation throughout. The test suite is extensive (>100 test files) covering API endpoints, domain logic, error handling, and edge cases. There are some maintenance risks related to tight coupling with SurrealDB and the externally-maintained Esperanto/content-core/podcast-creator libraries, but overall the codebase is actively maintained and of professional quality.