Cheerio is a fast, flexible, and elegant library for parsing and manipulating HTML and XML on the server side (Node.js). It provides a jQuery-like API for traversing and manipulating the DOM of parsed documents, designed for web scraping, data extraction, and HTML transformation tasks. The library abstracts away the underlying parser (htmlparser2 for speed, parse5 for standards compliance) via a factory pattern, offering a consistent Cheerio API regardless of parser choice. Key entry points include the main `load` function that converts HTML strings or buffers into a Cheerio instance, static methods like `html`, `xml`, `text`, and `parseHTML`, and utility functions for CSS handling, form serialization, and attribute manipulation. External dependencies include cheerio-select for CSS selector parsing, domutils for core DOM traversal, and optional HTTP fetching via undici for live URL parsing.
htmlparser2 is a fast, forgiving HTML/XML parser for Node.js and browsers, written in TypeScript. It provides a low-level streaming tokenizer (Tokenizer) that feeds events to a higher-level Parser, which handles tag matching, implicit closing, foreign content (SVG/MathML), and entity decoding. The library also includes Node.js WritableStream and Web Streams API WritableStream adapters for piping data directly into the parser. It is the foundation for cheerio and other DOM manipulation libraries. The codebase is well-structured, thoroughly tested, and actively maintained, with a focus on performance and spec compliance for real-world HTML.
This is a Deno/TypeScript library for extracting structured article data (title, content, author, date, image, etc.) from any URL or HTML string. It uses Mozilla's Readability for the core extraction, supplemented by custom metadata extraction from HTML meta tags, JSON-LD schemas, and link elements. The library provides a transformation system that allows per-site pre/post processing of HTML to improve extraction quality, and a flexible HTML sanitizer to control output tags and attributes. The architecture is modular, with clear separation of concerns: retrieval, metadata extraction, URL normalization, readability extraction, sanitization, and transformation. Entry points are `extract()` and `extractFromHtml()`, with optional custom fetch functions and parser options. The codebase is well-tested, actively maintained, and demonstrates solid engineering practices with minimal dependencies.
The ultimate-web-scraper repository is a comprehensive PHP toolkit for web scraping and related web interaction tasks. It provides a robust HTTP client (WebBrowser) that handles cookies, redirects, SSL/TLS, and proxy support, enabling reliable web requests. For HTML processing, it includes the TagFilter library, which offers high-performance parsing with CSS3 selector support, streaming capabilities for large files, and built-in XSS prevention. The toolkit also features WebSocket client and server implementations, a simple HTTP server, and utilities for asynchronous operations, DNS over HTTPS, and data streaming. Designed as a modular library, it allows developers to extract and reuse components like the HTTP client, HTML parser, or async helper in other projects with minimal dependencies.
sax-wasm is a high-performance SAX-style XML/HTML/JSX parser written in Rust and compiled to WebAssembly, with JavaScript/TypeScript bindings for Node.js and browsers. It provides streaming parsing of byte streams, emitting events for tags, attributes, text nodes, processing instructions, comments, CDATA, and doctype declarations. The parser is non-validating and focuses on raw speed, using SIMD instructions and optimized grapheme cluster handling. It includes a comprehensive TypeScript API with async generator support for streaming reads from ReadableStream. The repository contains the full Rust source, JS/TS bindings, build tooling (wasm-bindgen, wasm-opt), and extensive test suites covering XML, JSX, Angular templates, edge cases with multi-byte characters, and write-boundary splitting. While the parser is highly optimized and well-tested, the project appears to be in a stale maintenance state with no recent commits (last commit dated 2026-09-13, likely erroneous) and a small community following.
This repository is a collection of Kodi/XBMC addons focused on media streaming, playback, and extended functionality. It contains numerous plugin.video.* and plugin.audio.* addons that scrape or API-fetch content from various online sources (e.g., MEGOGO.NET, Kinozal.tv, YouTube, torrent sites) and integrate with Kodi's media player. The repository also includes utility modules for HTML parsing (html5lib, BeautifulSoup), JSON handling (simplejson), downloading, caching, and framework components (xbmcswift2). The architecture is a collection of largely independent addons, each with its own default.py entry point, shared libraries, and resource files. Data flows involve fetching metadata from external sources, parsing it into Kodi-compatible formats, and passing media URLs to Kodi's player. External dependencies include Kodi/XBMC APIs, vendored Python libraries (html5lib, beautifulsoup4, simplejson), and external web services/APIs.