Defuddle is a content extraction library that converts web pages into clean, readable Markdown with structured metadata (title, author, site, published date). It is the engine behind the Obsidian Web Clipper browser extension and also runs as a standalone CLI tool, an npm package, and a hosted API service on Cloudflare Workers. The architecture follows a modular pipeline: fetch HTML, detect site-specific extractors, score and select main content, remove clutter (nav, ads, sidebars), standardize complex elements (footnotes, math, code blocks, images), and convert to Markdown via Turndown. The library is designed as a more forgiving alternative to Mozilla's Readability, with support for mobile-style heuristics, schema.org metadata extraction, and extensive site-specific extractors for platforms like YouTube, Reddit, Twitter/X, ChatGPT, GitHub, and Gmail. The codebase is written in TypeScript and produces three bundles: core (browser, no extra dependencies), full (adds MathML-to-LaTeX conversion), and node (accepts any DOM implementation). The CLI uses Commander and supports file, URL, and stdin input with options for Markdown, JSON, frontmatter, and debug output. The hosted API runs on Cloudflare Workers with Stripe-based billing, rate limiting, and Durable Objects for stateful API key management. The test suite is exceptionally comprehensive, with hundreds of HTML fixtures and expected Markdown outputs covering edge cases from dozens of real-world websites, CMS patterns, and syntax highlighters. Key external dependencies include Turndown (Markdown conversion), linkedom (DOM parsing in Node.js), mathml-to-latex and Temml (math rendering), and various third-party APIs for site-specific extractors (YouTube InnerTube, FxTwitter, Reddit Atom feeds, Bilibili). The project maintains strict security practices: no innerHTML assignments outside a single utility function, URL sanitization, and XSS prevention through template-based HTML parsing. The build system uses webpack with aliases for math module switching and enforces bundle size budgets via CI scripts.
This is a PHP library that converts HTML strings to Markdown. It parses HTML into a DOMDocument, then recursively traverses the DOM tree, converting each node to its Markdown equivalent using a registry of converter classes. The architecture is clean: an `HtmlConverter` orchestrates the process, an `Environment` manages configuration and converter registration, and individual converter classes handle specific HTML elements (headers, links, lists, tables, etc.). The library supports extensive configuration options including stripping tags, preserving comments, customizing emphasis styles, and enabling table conversion. It provides both a programmatic API and a CLI binary. The code is well-structured with strict typing, interfaces, and comprehensive test coverage. It demonstrates solid software engineering practices and is actively maintained.
omni-article-markdown is a CLI tool and library that converts web articles from various sources into clean Markdown format. It employs a modular pipeline architecture: Reader fetches HTML (using requests or Playwright for JavaScript-heavy sites), Extractor isolates article content by removing boilerplate and ads, and Parser converts the cleaned HTML to Markdown with support for code blocks, tables, math equations, and GitHub Gists. The system uses a plugin-based architecture where site-specific readers and extractors are automatically discovered and loaded, allowing easy extension for new websites. Key features include SSL verification control, cookie persistence for sites requiring authentication (like Zhihu), and flexible output options (stdout or file saving).
Firecrawl is a production-grade web scraping, crawling, and structured data extraction platform designed to feed AI agents and LLMs with clean, markdown-formatted web data. The system exposes a multi-version REST API (v0, v1, v2) with endpoints for scraping single URLs, crawling entire sites, mapping site structures, searching the web, extracting structured JSON, monitoring page changes, and running autonomous AI agents. It supports multiple output formats including markdown, HTML, screenshots, PDF parsing, product/menu extraction, and branding detection. The architecture is a microservices-style monolith with a Node.js/Express API server at its core, backed by Redis for job queues and caching, PostgreSQL for persistent state, ClickHouse for analytics, Bigtable for job access logs, and GCS for blob storage. Background workers handle crawl orchestration, PDF processing, webhook delivery, and billing via RabbitMQ and FoundationDB queues. The platform integrates with external services for threat protection (Google Web Risk, Zscaler), billing (Autumn), search (SearXNG, DuckDuckGo, X Search), and LLM-based extraction (OpenAI, Anthropic). Nine official SDKs (Python, JavaScript/TypeScript, Go, Java, .NET, Ruby, PHP, Rust, Elixir) wrap the API with typed models, error handling, and async support, making it accessible across ecosystems. The codebase is extensive and well-tested, with over 2,000 files spanning controllers, scrapers, transformers, services, and utilities, but shows signs of technical drift through deprecated v0/v1 endpoints, multiple parallel implementations of similar logic (e.g., extract pipelines), and configuration sprawl across environment variables.
wp2hugo is a Go-based CLI tool for migrating WordPress XML exports to Hugo static sites. It processes WordPress content (posts, pages, custom post types) converting them to Hugo Markdown with appropriate front matter, handles media downloads, converts WordPress shortcodes and Gutenberg blocks to Hugo shortcodes, preserves metadata like GUID and taxonomies, and generates Hugo configuration files. The tool also includes a companion utility 'hugomanager' for site management tasks like backlink analysis, image optimization, and URL suggestions. The architecture is modular with clear separation between WordPress parsing, Hugo generation, media handling, and utility functions.