$ show extractus/article-extractor
This is a Deno/TypeScript library for extracting structured article data (title, content, author, date, image, etc.) from any URL or HTML string. It uses Mozilla's Readability for the core extraction, supplemented by custom metadata extraction from HTML meta tags, JSON-LD schemas, and link elements. The library provides a transformation system that allows per-site pre/post processing of HTML to improve extraction quality, and a flexible HTML sanitizer to control output tags and attributes. The architecture is modular, with clear separation of concerns: retrieval, metadata extraction, URL normalization, readability extraction, sanitization, and transformation. Entry points are `extract()` and `extractFromHtml()`, with optional custom fetch functions and parser options. The codebase is well-tested, actively maintained, and demonstrates solid engineering practices with minimal dependencies.
The LLM identified 5 files worth highlighting for reuse:
Provides a configurable HTML sanitizer (`cleanify`) that walks the DOM tree, removes disallowed tags, strips disallowed attributes, and enforces iframe domain whitelisting. It is self-contained (only depends on `linkedom` for parsing) and can be extracted into any project needing safe HTML output. The implementation handles edge cases like nested elements and attribute removal efficiently.
A collection of URL utility functions including validation, purification (removing tracking parameters from a curated blacklist), absolutification of relative URLs, normalization (making all links absolute and adding target=_blank), and best URL selection via string similarity to the article title. The tracking parameter removal is particularly well-curated and reusable in any scraping or URL processing pipeline.
Implements a registration and execution system for per-site transformations. Users can add transformations (pre/post functions) that match URL patterns, allowing site-specific DOM manipulation before or after readability extraction. The module is self-contained and can be reused as a lightweight middleware pattern in other extraction or web scraping tools.
Extracts metadata (author, date, description, image, type) from JSON-LD `<script>` tags. It validates the schema type against a list of known article/webpage types, parses the JSON, and extracts the relevant fields. The logic handles arrays, objects, and nested structures gracefully. This module is reusable for any project needing to parse JSON-LD metadata from HTML.
A heuristic date extraction function that searches the DOM for date patterns in `<time>` elements, elements with `datetime` attribute, itemprop attributes, and then falls back to scanning text content of paragraphs, spans, and divs. It handles multiple date formats and converts them to ISO strings. This is a self-contained utility that can be reused in any HTML parsing project needing date discovery.