$ show cubiclesoft/ultimate-web-scraper
The ultimate-web-scraper repository is a comprehensive PHP toolkit for web scraping and related web interaction tasks. It provides a robust HTTP client (WebBrowser) that handles cookies, redirects, SSL/TLS, and proxy support, enabling reliable web requests. For HTML processing, it includes the TagFilter library, which offers high-performance parsing with CSS3 selector support, streaming capabilities for large files, and built-in XSS prevention. The toolkit also features WebSocket client and server implementations, a simple HTTP server, and utilities for asynchronous operations, DNS over HTTPS, and data streaming. Designed as a modular library, it allows developers to extract and reuse components like the HTTP client, HTML parser, or async helper in other projects with minimal dependencies.
The LLM identified 3 files worth highlighting for reuse:
A robust HTML/XML parser that supports CSS3 selectors for efficient data extraction. It features a streaming mode for processing large files with minimal memory usage, a DOM-like node tree for manipulation, and built-in XSS prevention via configurable HTML purification. The parser handles malformed HTML gracefully and is self-contained with only a UTF utilities dependency.
A full-featured HTTP client that emulates web browser behavior, including automatic cookie handling, referer header management, user agent rotation, and redirect following. It supports SSL/TLS, proxy tunneling, and asynchronous operations via callbacks. The class encapsulates session state across requests, making it ideal for web scraping tasks that require maintaining login state or handling complex interactions.
Manages multiple asynchronous operations with concurrency limits using PHP's stream_select for I/O multiplexing. It provides a simple interface to register callbacks for readable/writable streams and handles resource cleanup automatically. This component is decoupled from specific protocols and can be reused for any task requiring non-blocking I/O, such as HTTP requests, WebSocket communication, or file operations.