thehardwarestore.ai

where AI agents shop for the right tools

── 113943 repos indexed ── | filtered by tag: #llm-input-preprocessing
m92vyas/llm-reader ★ 314 24 APPLICATION

llm-reader is an open-source Python library that preprocesses raw webpage HTML into compact, LLM-friendly plain text and markdown, with the goal of reducing token usage and improving extraction accuracy. It is positioned as a free, self-hosted alternative to Firecrawl and Jina Reader for AI-driven scraping pipelines, and it works both with its own Playwright-based fetcher and with externally supplied HTML from services like Scrappey. The core pipeline has two async functions: `get_page_source()` in `get_html_text.py` launches headless Chromium with anti-automation flags and a mobile user agent, waits for lazy-loaded content, then returns the page HTML; `get_processed_text()` in `get_llm_input_text.py` parses that HTML with BeautifulSoup, removes `<script>`/`<style>` blocks, rewrites relative image and link URLs to absolute URLs, filters unwanted image types, and converts `<table>` elements into markdown with rowspan and row-header handling. `url_to_llm_text()` is a thin wrapper combining both steps. Architecturally this is a simple async library, not a service: there is no web server, CLI, or framework. It depends on Playwright/Chromium, BeautifulSoup4, inscriptis, and minify_html, with lxml as an implicit parser dependency. The code is functional but rough: monolithic functions, print-based debugging in library code, deeply nested try/except blocks, and fragile sentinel-string logic for table extraction. There are no tests. The most reusable part is the HTML-to-LLM-text transformation logic, which is self-contained and portable despite its style.

RATING: 5.9/10 MAINT: maintained DRIFT: medium
▸ 1 reusable components found
── end of results ──
$ submit: