$ show scrapinghub/extruct
This repository is a Python library and command-line tool for extracting structured data from HTML documents. It supports multiple formats including W3C HTML Microdata, JSON-LD, Microformat, Open Graph, RDFa, and Dublin Core. The project provides a main extract function that runs all enabled extractors and returns the results in a dictionary, with an option to convert the output to a uniform JSON-LD-like structure for easier consumption. The architecture is modular, with each format handled by a separate extractor class. The main orchestration function in `_extruct.py` manages the extraction process, error handling, and result aggregation. The command-line interface allows users to extract data from a URL or file and output JSON directly. The project includes an extensive test suite with sample HTML files and expected outputs for each format, covering valid cases, edge cases, and invalid inputs. Key reusable components include the uniform output transformation module (`extruct/uniform.py`), which normalizes data from different formats into a common structure, and the error handling pattern used throughout the extraction pipeline. The project is actively maintained, with recent updates adding support for newer Python versions and maintaining compatibility with dependencies. Technical drift is low due to regular updates and a focus on compatibility.
The LLM identified 1 files worth highlighting for reuse:
This module provides a uniform output transformation that converts extracted data from various formats (Microdata, JSON-LD, etc.) into a common JSON-LD-like structure. It takes the raw extraction results (a dictionary format->list of items) and adds @context, @type, and normalizes property values. This pattern is reusable in any project needing to normalize heterogeneous structured data sources into a consistent format, demonstrating skill in handling data variability without tight coupling to specific extractors.