MarkItDown is a Python library and CLI tool that converts a wide variety of file formats (PDF, DOCX, PPTX, XLSX, HTML, CSV, JSON, EPUB, Outlook MSG, images, audio, YouTube, RSS, IPYNB, ZIP, and more) into Markdown. Its primary production use is preparing documents for LLM ingestion, agent workflows, and text analysis pipelines. The architecture is a monorepo containing the core `markitdown` package, an OCR extension (`markitdown-ocr`) that uses LLM Vision for image text extraction, an MCP server (`markitdown-mcp`) exposing conversion as a tool for AI agents, and a sample plugin for third-party converters. The core library uses a plugin system based on `importlib.metadata.entry_points` to register converters, each implementing a `DocumentConverter` base class with `accepts()`/`convert()` methods and a priority-based dispatch. The main entry points are the `MarkItDown` class (Python API), the `markitdown` CLI, and the MCP server's `convert_to_markdown` tool. Data flows from local files, streams, HTTP responses, or URIs through converter selection (by MIME type, extension, or content inspection) to produce Markdown output. External dependencies include many optional format-specific libraries (pdfminer.six, python-docx, python-pptx, openpyxl, etc.), external binaries (exiftool, ffmpeg), cloud APIs (Azure Document Intelligence, Azure Content Understanding), and an OpenAI-compatible LLM client for image captioning and OCR.
Excalidraw is an open-source virtual whiteboard and diagramming application, published both as a hosted PWA (excalidraw-app) and as an npm library (@excalidraw/excalidraw). The system provides an infinite canvas with hand-drawn-style rendering, supporting a wide range of shapes (rectangles, ellipses, arrows, lines, freedraw, text, sticky notes, images, frames, embeddables) and tools (laser pointer, eraser, bucket fill, lasso selection, shape recognition). It features real-time collaboration via WebSocket (socket.io) and Firebase, with end-to-end encryption, cursor sharing, and follow/unfollow. The architecture is a React-based single-page application with a modular monorepo structure: the core editor logic resides in packages/excalidraw, element manipulation in packages/element, geometry utilities in packages/math, shared constants and data structures in packages/common, and export helpers in packages/utils. State management uses Jotai atoms and a custom action registration system (ActionManager). The editor renders to an HTML5 canvas with a separate SVG overlay for interactive elements. Key entry points include packages/excalidraw/index.tsx (library export) and excalidraw-app/index.tsx (hosted app bootstrap). Data flows from user interactions through actions to element mutations, scene updates, and canvas re-rendering. External dependencies include Firebase (Firestore, Storage), socket.io, Radix UI, Jotai, roughjs, and various build tools. The codebase also includes an AI-assisted text-to-diagram feature (TTD dialog) that converts Mermaid syntax into Excalidraw elements via streaming SSE and auto-fix logic.
This repository is the openai/codex monorepo, which implements a local AI coding agent called Codex (also referred to as Codex CLI). It provides a terminal user interface (TUI), command-line interface (CLI), and a local app-server daemon that manages AI-driven coding sessions. The architecture is client-server: the TUI/CLI communicates with the local app-server via JSON-RPC over Unix sockets or WebSockets, while the exec-server handles sandboxed command execution with platform-specific security policies (macOS Seatbelt, Windows sandbox, Linux bubblewrap). The system is built entirely in Rust, with additional TypeScript and Python SDKs for programmatic access. It supports a rich extension ecosystem including plugins, skills, MCP servers, memories, and a guardian safety layer for approval workflows. The protocol layer is generated from Rust types into TypeScript schemas, ensuring type safety across the boundary. The codebase is well-structured, heavily tested, and follows modern Rust practices (async/await, trait-based abstractions, comprehensive error handling). It is not a data research project; there are no notebooks, CSV files, or ML pipelines.
Graphify is a CLI tool and library that extracts structural and semantic knowledge graphs from source code across 36+ programming languages. It parses files using tree-sitter grammars, builds a graph of nodes (functions, classes, modules, etc.) and edges (calls, imports, inheritance, etc.), performs community detection via Leiden/Louvain, and generates reports, wikis, and exports to formats like Obsidian, Neo4j, and FalkorDB. The system integrates with AI coding assistants through an MCP server (stdio/HTTP), skill files (SKILL.md, CLAUDE.md), and git hooks for automatic graph updates. Key entry points include `graphify extract`, `graphify query`, `graphify serve`, and `graphify install`. External dependencies include tree-sitter grammars, LLM APIs (OpenAI, Anthropic, Gemini, Ollama), and optional databases. The architecture is modular: per-language extractors in `graphify/extractors/`, graph assembly in `build.py`, clustering in `cluster.py`, and export in `export.py`. The codebase demonstrates strong engineering practices with extensive tests, atomic file writes, security validation (SSRF, path traversal), and cross-platform support.
TradingAgents is a multi-agent LLM framework for financial trading research and backtesting. It orchestrates specialized agents (analysts, researchers, debaters, trader, portfolio manager) through a LangGraph state machine to produce trading decisions. The system emphasizes point-in-time data isolation to prevent lookahead bias, supports multiple LLM providers and data vendors, and includes a comprehensive backtesting engine. It is structured as a Python library with a rich CLI interface. The architecture is a hybrid of a software application and a research framework. It demonstrates high craftsmanship in its data pipeline (vendor routing, symbol normalization, date windowing), LLM integration (capability registry, structured output fallback), and concurrency handling (atomic file writes). While the core abstractions are clean and reusable, the codebase is tightly coupled to the specific behaviors of its external dependencies (LLM APIs, Yahoo Finance, SEC EDGAR), making it sensitive to ecosystem drift. The extensive test suite provides good coverage for unit and integration scenarios, but the reliance on mocking means regressions in vendor adapters may go undetected without manual integration runs.
This is a large polyglot monorepo for WiFi-based human sensing (WiFi-DensePose) that turns commodity WiFi routers into spatial intelligence systems. It captures Channel State Information (CSI) from ESP32 nodes to detect presence, estimate 17-keypoint body pose, measure breathing and heart rate, and track movement through walls without cameras or wearables. The system integrates deeply with Home Assistant via MQTT and Matter, provides a HomeKit bridge, and exposes multiple surfaces: a Rust backend workspace (v2/crates) with 50+ crates for signal processing, privacy shielding (VEIL/BFLD), pose physics, and home automation; Python bindings (PyO3) for ML training and inference; TypeScript CLI and MCP tools for AI agent integration; ESP32-C6/S3 firmware with adaptive sensing; React Native mobile app; Lit-based dashboard; and Tauri desktop app. The architecture is event-driven with a streaming engine, coherence-gated privacy layers, and deterministic verification via SHA-256 golden files. The codebase is heavily ADR-driven with extensive test coverage including cross-language parity gates. While primarily an application, it also contains substantial research code (benchmarks, physics simulations, SOTA surveys) and is best described as a research-informed production platform.
Uptime Kuma is a self-hosted, open-source uptime monitoring application that periodically checks the availability and response time of services (HTTP, TCP, DNS, databases, Docker containers, and many more) and sends alerts via 90+ notification providers. The system is a monolithic Node.js/Express backend with a Vue 3 SPA frontend, using Socket.IO for real-time updates and SQLite or MariaDB for persistence. It provides status pages, maintenance windows, certificate/domain expiry monitoring, and a Prometheus metrics endpoint. The architecture follows a plugin pattern for monitor types and notification providers, both built on base classes, and includes a condition evaluation system for advanced alerting logic. The codebase is well-structured with clear separation of concerns, extensive tests (unit, integration, e2e), and active community contributions.
Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
rtk is a Rust CLI proxy and hook system designed to minimize LLM token consumption for AI coding agents by filtering and compressing the output of shell commands. It operates as a command wrapper that intercepts commands like git status, pytest, or cargo test before they reach an agent's context window, runs the real tool, and emits a compact, token-optimized summary while preserving diagnostic value. The tool includes a sophisticated architecture: a core runner with a 'never-worse' guard that ensures filtered output never exceeds raw output in size, tiered parsing (JSON → regex → passthrough) for dozens of ecosystems, a declarative TOML filter DSL compiled into the binary, and a SQLite-based recall store for content-addressed output retrieval. It also features a permission model for hooks that can allow, deny, or ask on command rewrites, and integrates with 18+ AI agents (Claude Code, Copilot, Cursor, etc.) via custom hook formats. The repo is a single Rust binary with subcommands for execution, analytics (rtk gain), and session discovery (rtk discover), supported by an extensive test suite with fixture-based golden tests. While not a data research project, it incorporates data-driven elements like token savings accounting and telemetry, but its dominant character is a high-craft CLI application.
Hoppscotch is an open-source, multi-platform API development ecosystem that serves as a full-featured alternative to Postman. It supports REST, GraphQL, WebSocket, SSE, Socket.IO, and MQTT protocols, with advanced features like pre-request and test scripting (sandboxed), environment variables, team collaboration, mock servers, published documentation, AI assistant, and extensive import/export (Postman, Insomnia, OpenAPI, HAR, cURL). The monorepo is organized into packages: a Vue 3 frontend (hoppscotch-common), a NestJS/GraphQL/Prisma backend (hoppscotch-backend), a Tauri desktop app (hoppscotch-desktop), a Node.js CLI (hoppscotch-cli), a Tauri agent for interceptor (hoppscotch-agent), a shared kernel abstraction layer (hoppscotch-kernel), versioned data schemas (hoppscotch-data), a JavaScript sandbox engine (hoppscotch-js-sandbox), and self-hosted web/admin panels. The architecture emphasizes functional programming with fp-ts, reactive state management via RxJS DispatchingStores, dependency injection with dioc, and a kernel/relay pattern that abstracts HTTP execution across platforms. The codebase is actively maintained with extensive test coverage and demonstrates high craftsmanship in areas like versioned data migration, sandboxed script execution, and cross-platform kernel design.
Impeccable is a design-quality and anti-pattern detection engine for front-end web projects, primarily targeting AI-generated UI. It provides a Rust-based CLI, a WASM browser extension core, and a distributed skill system that plugs into over 20 AI coding assistants (Claude, Cursor, Gemini, etc.). The system runs deterministic detection rules against HTML/CSS to flag common issues like poor contrast, typography problems, layout flaws, and 'AI slop' patterns, and also supports live browser iteration with session management, component review workflows, and comp-vs-build image comparison. All detection logic is implemented as a port of a JavaScript reference, with extensive vector-replay tests to maintain byte-for-byte behavioral parity. Architecturally, the repository is a large Rust workspace with 14+ crates handling distinct concerns: static HTML parsing and CSS cascade (`crates/html`), rule evaluation (`crates/foundation`, `crates/core`), browser automation via CDP (`crates/browser`), live-mode HTTP servers and session management (`crates/live`), and skill packaging/distribution (`crates/skills`, `scripts/`). A Chrome DevTools extension embeds the WASM core for in-page scanning. The project is actively maintained with respect to both the Rust core and the constantly shifting AI-harness integration surface, though that breadth introduces significant drift risk across duplicated skill assets and provider-specific configuration. The codebase demonstrates strong engineering discipline—clean abstractions, pervasive testing, and careful parity management—but the sheer scale and duplication in the skill/plugin tree reduces overall reusability.
OpenSpec is an AI-native spec-driven development CLI tool that provides a lightweight agreement layer between humans and AI coding assistants. It manages structured 'changes' with proposals, delta specs, designs, and tasks, supporting 30+ AI coding tools (Claude, Cursor, Devin, Copilot, etc.) by generating per-tool skills and slash commands. The CLI features a store system for cross-repo planning, an artifact graph for tracking change progress, shell completion generation, and a validation system. The codebase is well-structured with extensive test coverage, reusable components (artifact graph, markdown parser, command generation adapters, store foundation), and security-conscious patterns (atomic writes, prototype pollution guards). It is actively maintained with a large community (70k+ stars).
Docling is a production-grade Python library for converting diverse document formats (PDF, DOCX, PPTX, XLSX, HTML, LaTeX, EPUB, images, audio, video, email, EBCDIC, and more) into a unified, structured `DoclingDocument` representation. It is designed to serve as the document understanding layer for AI workflows, particularly Retrieval-Augmented Generation (RAG), agentic AI, and document extraction pipelines. The system provides a rich Python SDK, a Typer-based CLI (`docling convert`), a service client for remote conversion via `docling-serve`, and an MCP server. It supports multiple output formats including Markdown, JSON, HTML, DocTags, and plain text. Architecturally, Docling employs a modular pipeline system with pluggable backends for each input format and a stage-based processing model. The core pipeline (`StandardPdfPipeline`, `VlmPipeline`, `AsrPipeline`) orchestrates page-level operations (preprocessing, OCR, layout detection, table structure recognition, picture classification, reading order) and document-level enrichments (code/formula extraction, heading hierarchy, picture description). Inference engines are abstracted behind factory patterns supporting ONNX Runtime, HuggingFace Transformers, vLLM, MLX, KServe v2, and remote APIs. The library integrates with over 20 OCR engines and vision-language models, and includes a plugin system for extensibility. Key entry points include the `DocumentConverter` class, the CLI (`docling.cli.main`), and the service client (`docling.service_client`). External dependencies span PDF libraries (pypdfium2, docling-parse), ML frameworks (torch, transformers, onnxruntime), OCR engines (Tesseract, EasyOCR, RapidOCR), and cloud services (KServe, Azure, S3). The codebase is actively maintained with frequent releases, extensive test coverage, and a clear migration path from legacy components.
drawio-desktop is an Electron-based desktop application that wraps the core draw.io diagramming editor (included as a git submodule). It provides a fully offline diagramming experience with a security-first design: Content Security Policy prevents remote script execution, IPC is gated through contextBridge with sender validation, and file operations enforce path authorisation against symlink attacks. The application supports CLI-based batch export (PNG, PDF, SVG, HTML, XML) with a hand-rolled argument parser, per-window file watching, auto-update via electron-updater, and platform-specific builds for Windows (x64/arm64, MSI, NSIS, APPX), macOS (dmg, zip, Quick Look preview), and Linux (AppImage, deb, rpm, snap). The architecture is a monolithic Electron main process (src/main/electron.js) communicating with the renderer (drawio submodule) through a secure IPC bridge. The codebase demonstrates high craftsmanship in security-sensitive areas such as backup file writes using exclusive temp files and rename, CLI export symlink handling, and per-window file watch registry. It is actively maintained with recent commits and comprehensive CI/CD pipelines.
MiniMind is an educational open-source toolkit for training small language models entirely from scratch using native PyTorch. It provides a complete pipeline including custom BPE tokenizer training, pretraining, supervised fine-tuning (SFT), LoRA, RLHF (DPO), RLAIF (PPO/GRPO/CISPO), agentic reinforcement learning, and knowledge distillation. The system is designed to be accessible (low compute cost, ~3 RMB and 2 hours on a single GPU) and includes inference interfaces such as a CLI, Streamlit web UI, and an OpenAI-compatible API server. While primarily a software application for training and deploying LLMs, it also contains dataset preparation, evaluation benchmarks, and research experiments, giving it a hybrid character that leans toward application due to its focus on providing reusable training scripts and deployment tools.
Ultralytics is a production-grade computer vision library built around the YOLO model family. In production it provides a unified Python/CLI API for loading pretrained and custom models, running inference, training, validation, export, benchmarking, and object tracking across detection, instance and semantic segmentation, pose estimation, oriented bounding boxes, classification, and depth estimation. The architecture is a modular Python package: an engine layer (Model, BasePredictor, BaseTrainer, BaseValidator), a unified AutoBackend inference dispatcher supporting 20+ formats from PyTorch to TensorRT/ONNX/OpenVINO/CoreML/TFLite, a YAML-driven model configuration system, dataset loaders with caching and augmentation, pluggable experiment-tracking callbacks, and a family of ready-to-use solution classes for counting, heatmaps, queue management, and similar vision applications. The CLI entrypoint (`yolo`) and Python classes (`YOLO`, `SAM`, `RTDETR`, etc.) are the primary entry points, backed by active CI, Docker images, and extensive examples in Python, C++, and Rust.
This repository is the primary implementation of the Sway programming language and its toolchain, targeting the Fuel blockchain. It contains a full compiler pipeline (lexer, parser, AST, semantic analysis, IR with optimization passes, and code generation for the FuelVM), a package manager CLI (`forc`), a language server (`sway-lsp`), a code formatter (`swayfmt`), a debugger (`forc-debug`), a documentation generator (`forc-doc`), and a migration tool (`forc-migrate`). The standard library (`sway-lib-std`) provides essential blockchain primitives (assets, storage, crypto, etc.). The architecture is modular, with separate crates for the AST (`sway-ast`), IR (`sway-ir`), parsing (`sway-parse`), core compiler logic (`sway-core`), and error handling (`sway-error`). In production, developers use `forc` to build, test, and deploy Sway smart contracts, scripts, and predicates to the Fuel network. The LSP provides IDE features like completions, go-to-definition, and inlay hints. The formatter ensures consistent code style. The debugger enables step-through debugging of on-chain logic. The compiler supports multiple backends (FuelVM primary, EVM experimental) and includes advanced optimizations such as constant folding, dead code elimination, common subexpression elimination, and register allocation. Key entry points include `forc/src/main.rs` (CLI dispatcher), `sway-core/src/lib.rs` (compiler core), `sway-lsp/src/server.rs` (LSP server), and `swayfmt/src/lib.rs` (formatter). External dependencies include the Fuel protocol libraries (`fuel-core`, `fuel-vm`, `fuels-rs`), IPFS for package resolution, and standard Rust ecosystem crates. The repository is production-grade with extensive end-to-end tests, benchmarking infrastructure, and CI workflows.
Fuel Core is a production-grade Rust implementation of the Fuel blockchain node, serving as the execution layer for a modular rollup ecosystem. It operates as a full node with block production (PoA consensus via Redis leader lease), transaction pool, upgradable WASM-based execution, P2P networking (libp2p with gossipsub and request-response), an Ethereum relayer bridge, and a comprehensive GraphQL API for queries and transactions. The system uses a service-oriented architecture with clear port/adapter boundaries, allowing each subsystem (sync, compression, gas pricing, tx status management, etc.) to be independently tested and swapped. Key data flows include transaction submission through GraphQL or P2P gossip, verification by the txpool, inclusion in blocks by the producer, execution by the upgradable executor, and persistence to RocksDB. The codebase demonstrates high craftsmanship with clean abstractions, extensive integration and property-based testing, and a strong focus on performance and observability (Prometheus metrics, structured tracing). While it is primarily an application, it also includes research-grade components such as gas price algorithms (V0 and V1 with PD controllers) and block compression schemes.
PowerShell 7 (pwsh) is a production-grade cross-platform automation and configuration shell: a scripting language runtime, command engine, and host application. The repository contains the core engine assembly (System.Management.Automation), console host, global tool shim, hundreds of built-in cmdlets, provider infrastructure, remoting, formatting/serialization, and a meta-package SDK for embedding PowerShell in .NET applications. In production it executes scripts and commands, pipelines objects through cmdlets and providers, formats results for console or serialized transport, and supports both local runspaces and remote sessions over WinRM/WSMan and SSH. The primary entry points are ConsoleShell.Start() for the interactive host, the GlobalToolShim for the dotnet tool distribution, and the PowerShell.Create() hosting API used by embedded scenarios.