ProxyPool is a production-grade proxy pool application that automatically collects free HTTP/HTTPS proxies from over a dozen public sources, validates their availability, and stores them in Redis or SSDB. It provides a Flask RESTful API for consuming proxies (get, pop, all, count, delete) and uses APScheduler for periodic fetching and validation. The architecture is a monolith with two main processes: a scheduler that runs fetch/check cycles and an API server (Flask with Gunicorn on Linux). The codebase demonstrates strong software engineering practices: a plugin-based fetcher system with dynamic module loading and caching, a clean Proxy data model with JSON serialization, a decorator-based validator registration pattern, and a layered test suite (unit, API, integration) with fakeredis mocking. External dependencies include Redis/SSDB, APScheduler, Flask, requests, and lxml. The project is actively maintained with recent commits and a comprehensive test suite.
This is a Go-based video parsing tool that removes watermarks from videos across 20+ Chinese social media platforms (Douyin, Kuaishou, Bilibili, Weibo, etc.). It provides three interfaces: a CLI tool (using Cobra subcommands `serve`, `parse`, `id`, `version`), an HTTP API (Gin-based with v1 RESTful endpoints and legacy backward-compatible routes), and a reusable Go library. The architecture is a monolith with a clean separation: `parser/` package contains platform-specific parsers implementing strategy pattern interfaces, `cmd/` package handles CLI and HTTP server logic with a middleware stack (recovery, CORS, rate limiting, basic auth, logging), and `utils/` provides URL extraction. The system is stateless, uses no database, and relies on HTTP scraping and API calls to extract video metadata. Notable engineering craftsmanship includes a custom anti-crawling a_bogus signature generation for Douyin's native detail API (SM3 hash, RC4 encryption), a concurrent-safe proxy injection via atomic.Value, and a well-tested middleware stack with IP-based rate limiting using sync.Map and periodic cleanup. The codebase has comprehensive unit and integration tests, CI/CD pipelines, and Docker support.
Colly is a mature, high-performance web scraping framework for Go, providing a clean, callback-based API for crawling and extracting structured data from HTML and XML. It handles HTTP requests, session management, caching, robots.txt compliance, proxy rotation, rate limiting, and parallel scraping out of the box. The library is built around a Collector object that orchestrates configurable callbacks (OnRequest, OnResponse, OnHTML, OnXML, OnError, OnScraped) and supports chaining, context propagation, and request queueing for large-scale scraping jobs. The codebase is well-structured, thoroughly tested, and follows Go idioms, making it a reliable choice for production scraping tasks. It is not a data research project; it is a general-purpose scraping library.
This repository is an Android TV application (影視TV) that provides a comprehensive media center experience for streaming Video-on-Demand (VOD) and live television. It features a plugin-based content aggregation system using spiders written in Java, Python (via Chaquopy), and JavaScript (via QuickJS), allowing users to browse and play content from multiple external sources. The application supports multiple player engines (ExoPlayer and mpv), hardware/software decoding, subtitle and danmaku (bullet comment) overlays, DLNA casting (both sender and renderer), and a local HTTP server for remote control and file sharing. Architecturally, the project is a multi-module Gradle build with a shared core library (catvod) providing network utilities, cryptography, and data models. The main app module contains two UI tracks: 'leanback' for TV remote control navigation and 'mobile' for touch-based phone interaction, both sharing common logic under app/src/main. The system uses an event-driven architecture with EventBus, a Room-based SQLite database for persistence, and a ViewModel pattern for reactive UI updates. External content sources are integrated through a standardized Spider interface, with loaders for JAR, Python, and JavaScript plugins. Key features include a sophisticated playback controller with fallback policies, preloading, and history management; extensive audio/video effect pipelines (equalizer, loudness normalization, color adjustment); danmaku search and display; EPG (Electronic Program Guide) with timezone-aware parsing; and a comprehensive settings system covering player, decoding, subtitles, and network configuration. The project also includes a documentation website built with Next.js, providing developer guides for the spider API and local HTTP API.
EasySpider is a visual, no-code web scraping tool that combines an Electron desktop application, a Chrome browser extension, and a Python execution engine. Users design scraping workflows via a flowchart-based UI (Vue.js) where they define steps like opening pages, clicking elements, extracting data, looping, and conditional branching. The task definitions are saved as JSON graphs. The Python backend (ExecuteStage) loads these graphs and drives Chrome using Selenium with an undetected-chromedriver fork to evade bot detection. Extracted data can be written to CSV, Excel, JSON, or MySQL. The tool also includes OCR support (ddddocr), custom JavaScript/Python execution, and Cloudflare bypass capabilities. The architecture is a hybrid desktop app with a local HTTP server (Express) that spawns executor processes and communicates via WebSocket. The Chrome extension assists in element selection and XPath generation. While the tool is functional and popular (44k+ stars), the codebase exhibits significant duplication (e.g., identical task files across platform directories, duplicated Bootstrap and content scripts), hardcoded paths, monolithic executor scripts, and mixed-language strings. Several experimental modules (LLM, federated learning) are present but not integrated. The project is actively maintained with recent commits and packaging scripts, but technical debt and drift (Bootstrap 4, Vue 2, pinned Python dependencies) are moderate.
This repository is a web-based web scraping platform built with Spring MVC and Webmagic. It provides a user interface for configuring and managing web spiders (crawlers), scheduling crawl jobs via Quartz, processing crawled data with NLP (HanLP for keyword/summary/entity extraction), and storing results in Elasticsearch. The platform supports both static and Ajax crawling (via PhantomJS) and includes features for data search, CRUD operations, and export. The architecture follows a layered pattern with controllers, services, DAOs, and models, demonstrating separation of concerns. While the codebase exhibits solid architectural patterns and several reusable components, it shows signs of technical drift due to outdated dependencies (Spring 4.1.9, Webmagic 0.6.0, Bootstrap 4.0.0-alpha, Elasticsearch transport client 5.2.2) and deprecated technologies (PhantomJS).
This is a multi-module Spring Boot application for a Chinese-language novel reading and publishing platform (原创文å¦CMS). It comprises four interconnected modules: novel-admin (back-office management with Layui/Shiro RBAC), novel-front (user-facing reader/author portal with JWT auth), novel-crawl (automated content ingestion from external novel sites), and novel-common (shared entities, mappers, utilities). The system provides full CRUD for books, chapters, comments, users, and payments; templated front-end themes (blue, green, orange, dark); Alipay/WeChat integration; AI-assisted writing via Spring AI (SiliconFlow); and a MyBatis-based data layer with Sharding-JDBC for sharding. While functionally rich and actively maintained, the codebase suffers from significant static asset duplication, mixed javax/jakarta imports, and multiple legacy third-party front-end libraries bundled directly.
InfoSpider is a Windows-only desktop application that aggregates personal data from over 20 online services (e.g., GitHub, JD, Taobao, Alipay, Zhihu, Bilibili) using Selenium-based browser automation and a companion Chrome extension. The user launches a wxPython GUI, selects a data source, logs in via an automated Chrome window, and the tool scrapes orders, bills, contacts, posts, and other personal information, saving it as JSON files. Some spiders also generate HTML charts (word clouds, time-series plots) from the scraped data. The repository includes a separate, experimental PyTorch LSTM model for stock price prediction in `tests/DeepAnalysis/`, which is not integrated into the main tool. The codebase is a monolithic collection of spider scripts with a simple GUI wrapper, relying on outdated library versions (Selenium 3.141.0, pandas 1.0.1) and brittle automation patterns (hardcoded selectors, sleep-based waits). It has not seen active development since 2020 and carries significant technical drift and maintenance risks.
This repository contains only documentation and configuration files (README, CONTRIBUTING, issue templates, .gitignore) with no actual source code present. The README describes SpiderSuite as a cross-platform web crawler for security professionals, but the codebase itself is empty in this snapshot. No functional analysis is possible. The repository appears to be a stub or the source code is hosted elsewhere. Given the absence of any executable or library code, this repository provides no reusable software assets.