$ show Xewdy444/CF-Clearance-Scraper
CF-Clearance-Scraper is a Python CLI tool designed to solve Cloudflare challenges (JavaScript, managed, and interactive) and extract the cf_clearance cookie for bypassing anti-bot protections. The application uses Zendriver for headless browser automation, dynamically generates realistic Chrome user agents, handles proxy authentication via CDP events, and provides output formats including raw cookies, cURL/Wget/aria2 commands, or JSON storage. The architecture is monolithic with a single entry point (main.py) featuring clean separation of concerns: argument parsing, browser orchestration, challenge detection/solving, and output formatting. Key strengths include robust user agent fallback mechanics and proxy handling, though the Cloudflare-solving logic is tightly coupled to Zendriver and challenge-specific heuristics.
The LLM identified 2 files worth highlighting for reuse:
The Proxy class provides an isolated, reusable component for parsing and representing proxy server URLs. It includes a class method `from_url` to instantiate from a URL string and a property `url` to retrieve the proxy URL without authentication. This demonstrates clean encapsulation of proxy logic with zero external dependencies beyond the standard library, making it easily extractable for any project requiring proxy configuration handling.
The `get_chrome_user_agent` function generates a realistic, up-to-date Chrome user agent string by first attempting local version detection via `chrome_version` and falling back to Google's version history API if detection fails. This showcases a robust, non-obvious pattern for maintaining effective browser fingerprints in web scraping to avoid bot detection. The function is self-contained (aside from its declared dependencies) and can be reused in any HTTP automation project requiring credible user agent rotation.