Loofah is a Ruby library for manipulating and transforming HTML/XML documents and fragments, built on top of Nokogiri. In production it functions primarily as the HTML sanitization engine behind Rails (via rails-html-sanitizer) and countless other Ruby web applications: it parses untrusted user-supplied HTML and scrubs out unsafe tags, attributes, URLs, CSS properties, and protocols while preserving safe markup. The library combines a general-purpose transformation framework (custom 'scrubbers') with a specific html5lib-derived sanitizer safelist covering HTML, MathML, and SVG elements, ARIA attributes, CSS properties/functions, and URI protocols. The codebase deliberately supports both HTML4 (libxml2) and HTML5 (gumbo) parsing paths, and is the de facto standard for server-side HTML sanitization in the Ruby ecosystem.
This repository is a pure, generic microformats2 parser for PHP that makes HTML as easy to consume as a JSON API. It provides both functional and object-oriented interfaces for parsing microformats2 structured data from HTML documents. The parser handles all standard microformats2 property types (p-, u-, dt-, e-), implements implied property parsing, resolves relative URLs according to HTML and URI specifications, and includes backward compatibility for classic microformats markup. It features a robust test suite and follows PSR-0 autoloading standards.
MechanicalSoup is a Python library for automating interaction with websites. It simulates a web browser by automatically storing and sending cookies, following redirects, and submitting forms. It does not execute JavaScript, making it a lightweight alternative to Selenium for sites that rely on server-side logic. The library is built on top of Requests for HTTP sessions and BeautifulSoup for HTML parsing and manipulation. The architecture provides two main classes: Browser (low-level, stateless) and StatefulBrowser (high-level, stateful). StatefulBrowser is the recommended interface, maintaining the current page, URL, and selected form as internal state. Users can open URLs, follow links, select forms, fill in fields, and submit. The Form class handles form manipulation including text inputs, checkboxes, radio buttons, select dropdowns, textareas, and file uploads. It also correctly implements HTML form submission semantics such as disabled controls, fieldset disabling, and multipart encoding. Key entry points include StatefulBrowser.open(), select_form(), submit_selected(), follow_link(), and download_link(). The library depends on requests, beautifulsoup4, and lxml. It is designed as a reusable library, not a standalone application, and is typically installed via pip and imported into user scripts. The codebase is well-tested with 100% coverage, has comprehensive documentation, and follows semantic versioning.
Colly is a mature, high-performance web scraping framework for Go, providing a clean, callback-based API for crawling and extracting structured data from HTML and XML. It handles HTTP requests, session management, caching, robots.txt compliance, proxy rotation, rate limiting, and parallel scraping out of the box. The library is built around a Collector object that orchestrates configurable callbacks (OnRequest, OnResponse, OnHTML, OnXML, OnError, OnScraped) and supports chaining, context propagation, and request queueing for large-scale scraping jobs. The codebase is well-structured, thoroughly tested, and follows Go idioms, making it a reliable choice for production scraping tasks. It is not a data research project; it is a general-purpose scraping library.
This repository is a Python library and command-line tool for extracting structured data from HTML documents. It supports multiple formats including W3C HTML Microdata, JSON-LD, Microformat, Open Graph, RDFa, and Dublin Core. The project provides a main extract function that runs all enabled extractors and returns the results in a dictionary, with an option to convert the output to a uniform JSON-LD-like structure for easier consumption. The architecture is modular, with each format handled by a separate extractor class. The main orchestration function in `_extruct.py` manages the extraction process, error handling, and result aggregation. The command-line interface allows users to extract data from a URL or file and output JSON directly. The project includes an extensive test suite with sample HTML files and expected outputs for each format, covering valid cases, edge cases, and invalid inputs. Key reusable components include the uniform output transformation module (`extruct/uniform.py`), which normalizes data from different formats into a common structure, and the error handling pattern used throughout the extraction pipeline. The project is actively maintained, with recent updates adding support for newer Python versions and maintaining compatibility with dependencies. Technical drift is low due to regular updates and a focus on compatibility.