$ show jhao104/proxy_pool
ProxyPool is a production-grade proxy pool application that automatically collects free HTTP/HTTPS proxies from over a dozen public sources, validates their availability, and stores them in Redis or SSDB. It provides a Flask RESTful API for consuming proxies (get, pop, all, count, delete) and uses APScheduler for periodic fetching and validation. The architecture is a monolith with two main processes: a scheduler that runs fetch/check cycles and an API server (Flask with Gunicorn on Linux). The codebase demonstrates strong software engineering practices: a plugin-based fetcher system with dynamic module loading and caching, a clean Proxy data model with JSON serialization, a decorator-based validator registration pattern, and a layered test suite (unit, API, integration) with fakeredis mocking. External dependencies include Redis/SSDB, APScheduler, Flask, requests, and lxml. The project is actively maintained with recent commits and a comprehensive test suite.
The LLM identified 8 files worth highlighting for reuse:
Base class for all proxy fetchers. Defines a plugin contract (name, url, enabled attributes, fetch() method) and provides shared utilities like parseProxiesFromText (regex-based IP:port extraction) and yieldUniqueProxies (deduplication generator). This pattern allows adding new proxy sources by simply creating a new file in fetcher/sources/ that inherits from BaseFetcher, with zero configuration changes. The dynamic discovery and caching in helper/fetch.py makes it a hot-reloadable plugin system.
Clean Proxy data model with properties for proxy string, fail_count, region, source, check_count, last_status, last_time, and https flag. Provides to_dict, to_json serialization and a createFromJson classmethod for deserialization. The source property is stored as a list internally and joined with '/' on access, enabling additive source tracking via add_source(). This model is self-contained and could be reused in any proxy-related project.
A metaclass-based Singleton implementation that stores instances in a class-level dictionary. Simple, thread-safe (for CPython due to GIL), and easy to reset for testing (as done in conftest.py). The pattern is reusable across any Python project needing a single instance.
A descriptor that implements lazy property evaluation: on first access, it calls the function, caches the result as an instance attribute, and returns it. Used extensively in ConfigHandler to defer environment variable reads. This is a clean, reusable utility for any Python project.
Contains _discover_fetchers() which dynamically scans the fetcher/sources/ directory, imports or reloads modules based on file modification time (mtime), and returns all enabled BaseFetcher subclasses not in the exclude list. This enables hot-reloadable plugin discovery without restarting the scheduler. The module caching avoids repeated imports and handles file deletion cleanup. A sophisticated pattern for extensible systems.
Implements a decorator-based validator registration system via ProxyValidator classmethods (addPreValidator, addHttpValidator, addHttpsValidator). Validators are stored in class-level lists and executed in order. This allows users to add custom validation logic by simply decorating a function, without modifying core code. The pattern is clean and extensible.
A reusable HTTP client wrapper with automatic retry, random User-Agent rotation, and lxml HTML tree parsing. Provides get() and post() methods with configurable retry_time and retry_interval. The tree property lazily parses HTML content using lxml. The json property safely handles JSON parsing errors. This class is used by all fetchers and validators, and could be extracted as a standalone utility.
A well-crafted bash service management script supporting start (foreground/background), stop, restart, and status commands. Uses PID file for process tracking, graceful shutdown with SIGTERM followed by SIGKILL, and color-coded output. Designed for both container (foreground) and traditional (background) deployments. Demonstrates solid shell scripting practices.