Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
This repository is a comprehensive, async-first CLI application for downloading content from Douyin (TikTok China) without watermarks. It also powers a desktop application (Douzy) via a shared Python backend and offers an optional FastAPI REST server. The architecture follows a modular design: URL parsing dispatches to concrete downloader classes (video, music, live, user, mix) via a factory pattern; user-mode downloads employ a strategy pattern for posts, likes, mixes, and collections. Heavy emphasis is placed on anti-bot circumvention—X-Bogus and A-Bogus signature generation, browser fallback via Playwright, and risk-control retry logic—reflecting the adversarial nature of Douyin's API. The codebase is well-structured with distinct layers (auth, config, core, storage, control, utils) and extensive test coverage (>80 test modules). It demonstrates strong engineering practices: async I/O throughout (aiohttp, aiosqlite, aiofiles), configurable concurrency with rate limiting and retry backoff, incremental downloads via disk and database checks, structured progress reporting, and manifest logging. However, the project shows signs of technical drift: some dependencies are out of sync between requirements.txt and pyproject.toml, bundled msToken configuration may become stale, and the desktop-sidecar integration introduces coupling risks. Despite these, the code is actively maintained and functionally robust for its specific domain.
This repository is a collection of heavily obfuscated Node.js scripts designed to automate various JD.com (Jingdong) activities such as check-ins, bean collection, farm tasks, lotteries, and coupon management. The scripts are intended to run on the Qinglong (青龙) panel, a cron-based task scheduler. They rely on environment variables for configuration (cookies, notification settings, proxy) and use custom obfuscated signing algorithms to interact with JD's APIs. The codebase is tightly coupled to the JD ecosystem and the Qinglong platform, with no tests, no dependency pinning, and minimal documentation. The heavy obfuscation makes maintenance and reuse extremely difficult, and the reliance on external JD APIs introduces high technical drift risk. While the scripts are functional for their intended purpose, the overall software quality is low due to obfuscation, duplication, and lack of engineering best practices.