$ show kepano/defuddle
Defuddle is a content extraction library that converts web pages into clean, readable Markdown with structured metadata (title, author, site, published date). It is the engine behind the Obsidian Web Clipper browser extension and also runs as a standalone CLI tool, an npm package, and a hosted API service on Cloudflare Workers. The architecture follows a modular pipeline: fetch HTML, detect site-specific extractors, score and select main content, remove clutter (nav, ads, sidebars), standardize complex elements (footnotes, math, code blocks, images), and convert to Markdown via Turndown. The library is designed as a more forgiving alternative to Mozilla's Readability, with support for mobile-style heuristics, schema.org metadata extraction, and extensive site-specific extractors for platforms like YouTube, Reddit, Twitter/X, ChatGPT, GitHub, and Gmail. The codebase is written in TypeScript and produces three bundles: core (browser, no extra dependencies), full (adds MathML-to-LaTeX conversion), and node (accepts any DOM implementation). The CLI uses Commander and supports file, URL, and stdin input with options for Markdown, JSON, frontmatter, and debug output. The hosted API runs on Cloudflare Workers with Stripe-based billing, rate limiting, and Durable Objects for stateful API key management. The test suite is exceptionally comprehensive, with hundreds of HTML fixtures and expected Markdown outputs covering edge cases from dozens of real-world websites, CMS patterns, and syntax highlighters. Key external dependencies include Turndown (Markdown conversion), linkedom (DOM parsing in Node.js), mathml-to-latex and Temml (math rendering), and various third-party APIs for site-specific extractors (YouTube InnerTube, FxTwitter, Reddit Atom feeds, Bilibili). The project maintains strict security practices: no innerHTML assignments outside a single utility function, URL sanitization, and XSS prevention through template-based HTML parsing. The build system uses webpack with aliases for math module switching and enforces bundle size budgets via CI scripts.
The LLM identified 5 files worth highlighting for reuse:
Builds nested blockquote HTML from flat comment data arrays. Used by multiple extractors (Reddit, Hacker News, Mastodon) to render threaded conversations. Isolated utility with no internal imports beyond types, making it directly reusable in any project that needs to convert flat comment lists into hierarchical HTML.
Formats timestamped transcript segments into readable text with optional chapter markers and speaker labels. Used by the YouTube extractor to produce clean, structured transcripts from raw segment data. Self-contained with clear input/output contracts, reusable for any video or audio transcript processing.
ContentScorer class that evaluates DOM elements for likelihood of being main content based on text density, link density, and navigation indicators. Provides static methods like scoreElement, scoreAndRemove, isLikelyContent, and scoreNonContentBlock. This core algorithm is the heart of content detection and can be extracted into any content extraction pipeline.
Detects and standardizes footnotes from 15+ CMS patterns (WordPress, Google Docs, MediaWiki, O'Reilly, Texinfo, etc.) into a consistent format with backlinks. Demonstrates deep pattern matching and DOM traversal skills. The handler is self-contained and could be reused in any HTML-to-Markdown converter that needs robust footnote support.
Base class for site-specific conversation extractors (ChatGPT, Claude, Gemini, Grok). Provides shared logic for rendering messages, footnotes, and metadata. Implements the template method pattern with canExtract, extract, extractAsync, and prefersAsync lifecycle methods. Demonstrates clean separation of concerns and is reusable for building new conversation extractors.