$ show m92vyas/llm-reader
This repository provides a Python library for converting web pages into text optimized for Large Language Model (LLM) input. It fetches HTML content using Playwright (headless browser) and processes it to extract meaningful text while preserving structural information like tables, links, and images in a text format. The core processing logic converts HTML tables to markdown, handles image and link preservation (or removal), and strips unnecessary tags (script, style) to reduce noise. The library is designed as a preprocessing step before feeding web content to LLMs for tasks such as data extraction or summarization. It includes three main modules: HTML fetching, HTML processing, and a combined convenience function.
The LLM identified 1 files worth highlighting for reuse:
This module contains the core logic for processing HTML into LLM-ready text. It handles conversion of HTML tables to markdown format (including support for row headers and captions), processes image and anchor tags (either converting them to text URLs or removing them), and removes unnecessary tags (script, style, and custom tags). The function is self-contained, relying only on standard HTML parsing libraries (BeautifulSoup, re, urllib, minify_html, inscriptis), and demonstrates non-obvious skill in handling edge cases such as row spans in tables and nested structures. It can be extracted and used in any project requiring HTML-to-text conversion for LLM input.