$ show wujunwei928/parse-video
This is a Go-based video parsing tool that removes watermarks from videos across 20+ Chinese social media platforms (Douyin, Kuaishou, Bilibili, Weibo, etc.). It provides three interfaces: a CLI tool (using Cobra subcommands `serve`, `parse`, `id`, `version`), an HTTP API (Gin-based with v1 RESTful endpoints and legacy backward-compatible routes), and a reusable Go library. The architecture is a monolith with a clean separation: `parser/` package contains platform-specific parsers implementing strategy pattern interfaces, `cmd/` package handles CLI and HTTP server logic with a middleware stack (recovery, CORS, rate limiting, basic auth, logging), and `utils/` provides URL extraction. The system is stateless, uses no database, and relies on HTTP scraping and API calls to extract video metadata. Notable engineering craftsmanship includes a custom anti-crawling a_bogus signature generation for Douyin's native detail API (SM3 hash, RC4 encryption), a concurrent-safe proxy injection via atomic.Value, and a well-tested middleware stack with IP-based rate limiting using sync.Map and periodic cleanup. The codebase has comprehensive unit and integration tests, CI/CD pipelines, and Docker support.
The LLM identified 4 files worth highlighting for reuse:
Provides a resty HTTP client factory with proxy injection via atomic.Value. The InitProxy function validates proxy URL at startup and stores it atomically, while newClient() creates resty clients with the proxy pre-configured. This pattern solves the problem of injecting proxy configuration into 33+ independent resty.New() call points across 25 parsers without using global environment variables (which would affect Gin's own HTTP traffic). The atomic.Value ensures lock-free concurrent safety for a write-once-read-many scenario. This component is isolated (imports only resty and net/url) and can be extracted into any Go project needing configurable HTTP clients with proxy support.
Defines a unified API response format with success/error envelopes and semantic error codes (MISSING_PARAMETER, UNSUPPORTED_URL, PARSE_FAILED, etc.). Provides sendSuccess and sendError helper functions that produce consistent JSON responses. The classifyParseError function maps all parser errors to a single 422 PARSE_FAILED category. This pattern is reusable in any Gin-based API project and is isolated (imports only gin and net/http).
Implements a complete middleware stack for a Gin HTTP server: recovery (panic → 500), CORS with whitelist support, IP-based rate limiting using golang.org/x/time/rate with sync.Map for visitor tracking and a background goroutine for periodic cleanup of stale entries, request logging to stderr, and custom Basic Auth that returns v1 error format. The rate limiter uses RemoteAddr (not X-Forwarded-For) to prevent IP spoofing, and supports exempt paths/prefixes. This component is well-tested with table-driven tests and demonstrates non-obvious skill in concurrent map management and rate limit design. It depends only on gin, x/time/rate, and the internal response.go package.
Implements the a_bogus signature generation required by Douyin's native web detail API. This is a complex anti-crawling mechanism involving SM3 double-hashing, custom RC4 encryption, and a base64-like encoding with a custom alphabet. The component includes a full SM3 hash implementation (Chinese national standard), a deterministic a_bogus generator that takes query string, HTTP method, timestamps, and random seeds, and a fetcher that constructs the signed request URL. This demonstrates non-obvious cryptographic and reverse-engineering skill. While coupled to the Douyin parser, the a_bogus generation logic itself is isolated and could be extracted for any project needing to interact with Douyin's API.