$ show caol64/omni-article-markdown
omni-article-markdown is a CLI tool and library that converts web articles from various sources into clean Markdown format. It employs a modular pipeline architecture: Reader fetches HTML (using requests or Playwright for JavaScript-heavy sites), Extractor isolates article content by removing boilerplate and ads, and Parser converts the cleaned HTML to Markdown with support for code blocks, tables, math equations, and GitHub Gists. The system uses a plugin-based architecture where site-specific readers and extractors are automatically discovered and loaded, allowing easy extension for new websites. Key features include SSL verification control, cookie persistence for sites requiring authentication (like Zhihu), and flexible output options (stdout or file saving).
The LLM identified 4 files worth highlighting for reuse:
A generic plugin loader that dynamically discovers and instantiates classes from specified packages using importlib and pkgutil. It provides type-safe plugin instantiation with zero internal dependencies, making it highly reusable across Python projects. The pattern demonstrates advanced Python import system usage and eliminates the need for central registry maintenance.
A collection of well-crafted utility functions for text processing, URL handling, and metadata extraction. Includes functions for collapsing spaces, handling invisible Unicode characters, converting case formats, extracting OpenGraph/canonical metadata, and cleaning text. Most functions have minimal dependencies (primarily standard library) and solve recurring data cleaning problems with attention to edge cases like directional characters and zero-width spaces.
Manages Playwright lifecycle including automatic dependency installation, browser launching, and stealth page creation with anti-bot countermeasures. Handles Chromium binary installation, stealth script injection, and proper resource cleanup through context managers. Demonstrates sophisticated dependency management and browser automation patterns that can be reused in other web scraping or testing projects.
Cross-platform storage utility that handles configuration file persistence with automatic migration from legacy storage locations (~/.ommimd to platform-appropriate directories). Works seamlessly on Windows, macOS, and Linux by detecting environment variables and using appropriate path conventions. Includes atomic file operations and error handling for robust JSON-based storage of application state.