$ show NaiboWang/EasySpider
EasySpider is a visual, no-code web scraping tool that combines an Electron desktop application, a Chrome browser extension, and a Python execution engine. Users design scraping workflows via a flowchart-based UI (Vue.js) where they define steps like opening pages, clicking elements, extracting data, looping, and conditional branching. The task definitions are saved as JSON graphs. The Python backend (ExecuteStage) loads these graphs and drives Chrome using Selenium with an undetected-chromedriver fork to evade bot detection. Extracted data can be written to CSV, Excel, JSON, or MySQL. The tool also includes OCR support (ddddocr), custom JavaScript/Python execution, and Cloudflare bypass capabilities. The architecture is a hybrid desktop app with a local HTTP server (Express) that spawns executor processes and communicates via WebSocket. The Chrome extension assists in element selection and XPath generation. While the tool is functional and popular (44k+ stars), the codebase exhibits significant duplication (e.g., identical task files across platform directories, duplicated Bootstrap and content scripts), hardcoded paths, monolithic executor scripts, and mixed-language strings. Several experimental modules (LLM, federated learning) are present but not integrated. The project is actively maintained with recent commits and packaging scripts, but technical debt and drift (Bootstrap 4, Vue 2, pinned Python dependencies) are moderate.
The LLM identified 3 files worth highlighting for reuse:
A self-contained module that patches the ChromeDriver binary to evade bot detection. It replaces detection-related strings in the binary, caches the patched version, and provides a custom Chrome class that auto-downloads and patches the driver. This pattern is reusable for any Selenium-based automation project that needs to bypass Cloudflare or similar anti-bot measures.
Contains a safe template substitution function (`replace_field_values`) that expands `Field["name"]` placeholders using `repr()` and `json.dumps()` to prevent code injection in `eval()` and `JS()` calls. Also includes a reusable `myMySQL` class for config-driven database writes with typed table creation and deduplication. These utilities are isolated and can be extracted for other data pipeline projects.
A local HTTP server (Express) that provides REST endpoints for task CRUD, execution management, and file upload. It implements a `safeHandler` wrapper that catches exceptions and returns 500 without crashing, and a CORS allowlist that only permits local origins. This pattern is a good reference for building secure local control servers in desktop applications.