Defuddle is a content extraction library that converts web pages into clean, readable Markdown with structured metadata (title, author, site, published date). It is the engine behind the Obsidian Web Clipper browser extension and also runs as a standalone CLI tool, an npm package, and a hosted API service on Cloudflare Workers. The architecture follows a modular pipeline: fetch HTML, detect site-specific extractors, score and select main content, remove clutter (nav, ads, sidebars), standardize complex elements (footnotes, math, code blocks, images), and convert to Markdown via Turndown. The library is designed as a more forgiving alternative to Mozilla's Readability, with support for mobile-style heuristics, schema.org metadata extraction, and extensive site-specific extractors for platforms like YouTube, Reddit, Twitter/X, ChatGPT, GitHub, and Gmail. The codebase is written in TypeScript and produces three bundles: core (browser, no extra dependencies), full (adds MathML-to-LaTeX conversion), and node (accepts any DOM implementation). The CLI uses Commander and supports file, URL, and stdin input with options for Markdown, JSON, frontmatter, and debug output. The hosted API runs on Cloudflare Workers with Stripe-based billing, rate limiting, and Durable Objects for stateful API key management. The test suite is exceptionally comprehensive, with hundreds of HTML fixtures and expected Markdown outputs covering edge cases from dozens of real-world websites, CMS patterns, and syntax highlighters. Key external dependencies include Turndown (Markdown conversion), linkedom (DOM parsing in Node.js), mathml-to-latex and Temml (math rendering), and various third-party APIs for site-specific extractors (YouTube InnerTube, FxTwitter, Reddit Atom feeds, Bilibili). The project maintains strict security practices: no innerHTML assignments outside a single utility function, URL sanitization, and XSS prevention through template-based HTML parsing. The build system uses webpack with aliases for math module switching and enforces bundle size budgets via CI scripts.
This is a Deno/TypeScript library for extracting structured article data (title, content, author, date, image, etc.) from any URL or HTML string. It uses Mozilla's Readability for the core extraction, supplemented by custom metadata extraction from HTML meta tags, JSON-LD schemas, and link elements. The library provides a transformation system that allows per-site pre/post processing of HTML to improve extraction quality, and a flexible HTML sanitizer to control output tags and attributes. The architecture is modular, with clear separation of concerns: retrieval, metadata extraction, URL normalization, readability extraction, sanitization, and transformation. Entry points are `extract()` and `extractFromHtml()`, with optional custom fetch functions and parser options. The codebase is well-tested, actively maintained, and demonstrates solid engineering practices with minimal dependencies.
This repository contains the source code for SimpRead (įŽæĻ), a browser extension for Chrome and Firefox that provides an immersive reading mode, focus mode, annotation, and later-reading capabilities. The extension extracts article content from web pages, applies customizable typography and themes, and supports exporting to numerous services including Markdown, PDF, EPUB, Kindle, email, and cloud storage (Dropbox, WebDAV, Notion, Evernote, etc.). It includes a rules-based site adaptation engine (defined in `website_list.json`) to handle hundreds of specific websites, a plugin system for user-installed scripts, and a notice/focus system to manage reading sessions. The architecture follows a typical browser extension pattern: a background script (`background.js`) manages storage, OAuth, context menus, and cross-tab communication; content scripts (`contentscripts.js`) inject reading and focus modes into pages; and multiple options pages (`options/`) provide configuration, custom CSS, site management, and notice center. The UI is built with React 0.14.8 and a custom Material Design UI kit (`mduikit`), with heavy use of jQuery for DOM manipulation. State is managed through a central storage service (`storage.js`) that provides reactive reads/writes and versioned migration. External integrations are implemented as service classes with OAuth flows, and a message-action pattern coordinates between isolated script contexts. The codebase demonstrates a strong feature set but suffers from significant technical debt: very old dependencies (webpack 1, React 0.14), deprecated lifecycle patterns, extensive global DOM manipulation mixed with React, hardcoded CDN links and provider secrets, and comments indicating hacky browser workarounds. The site adaptation rule set is large but fragile, relying on exact CSS selectors and inline JavaScript that break when websites change. Despite these issues, the project provides a wealth of reusable patterns, particularly in its custom UI components, notification system, and storage abstraction.
SingleFileZ is a browser extension (Chrome/Firefox/Edge) and CLI tool that saves complete web pages as self-extracting HTML/ZIP polyglot files. The core output format embeds a ZIP archive containing all page resources (HTML, CSS, images, fonts, frames) inside an HTML wrapper with a JavaScript bootstrap that extracts and displays the page without external dependencies. The extension offers extensive configuration including multiple profiles, URL-based auto-save rules, cloud uploads (Google Drive, Dropbox, GitHub, WebDAV), an annotation editor for notes/highlights/cut operations, batch saving, and file name templating. Architecturally, the project is a multi-browser extension with background scripts orchestrating page capture, content scripts handling in-page UI and resource collection, and a task queue for managing parallel saves. It relies heavily on message passing between contexts and includes custom infrastructure: a binary serialization format (YABSON) for efficient data transfer, a full ZIP read/write library with support for encryption and streaming, and a fork of Mozilla's Readability.js for article extraction. The CLI component uses Docker to run headlessly, delegating to the `single-filez-cli` npm package. The codebase is large and feature-rich but shows signs of maintainability strain: core modules are distributed as minified build artifacts in `lib/`, the README states SingleFileZ has been merged into its parent project SingleFile (suggesting this repo is superseded), and it relies on Manifest v2 which is being phased out by browsers. The custom polyfills and browser-specific workarounds add complexity. Despite this, several components (YABSON serialization, the ZIP library, cloud upload abstractions) are well-isolated and could be reused independently with minimal modification.