$ show jiji262/douyin-downloader
This repository is a comprehensive, async-first CLI application for downloading content from Douyin (TikTok China) without watermarks. It also powers a desktop application (Douzy) via a shared Python backend and offers an optional FastAPI REST server. The architecture follows a modular design: URL parsing dispatches to concrete downloader classes (video, music, live, user, mix) via a factory pattern; user-mode downloads employ a strategy pattern for posts, likes, mixes, and collections. Heavy emphasis is placed on anti-bot circumvention—X-Bogus and A-Bogus signature generation, browser fallback via Playwright, and risk-control retry logic—reflecting the adversarial nature of Douyin's API. The codebase is well-structured with distinct layers (auth, config, core, storage, control, utils) and extensive test coverage (>80 test modules). It demonstrates strong engineering practices: async I/O throughout (aiohttp, aiosqlite, aiofiles), configurable concurrency with rate limiting and retry backoff, incremental downloads via disk and database checks, structured progress reporting, and manifest logging. However, the project shows signs of technical drift: some dependencies are out of sync between requirements.txt and pyproject.toml, bundled msToken configuration may become stale, and the desktop-sidecar integration introduces coupling risks. Despite these, the code is actively maintained and functionally robust for its specific domain.
The LLM identified 4 files worth highlighting for reuse:
An async token-bucket rate limiter with configurable rate and burst, plus random jitter. It is fully self-contained, imported by downloaders and API clients to throttle requests. The implementation uses an asyncio lock and a simple linear token model, making it easy to extract into any async Python project requiring rate-limited third-party API calls.
An adjustable-size async worker pool that manages concurrent download tasks. It supports dynamic max_workers changes, safe cancellation (CancelledError handling), and a release method that is explicitly synchronous to avoid event loop issues. This pattern is reusable for any async batch processing where concurrency must be controlled and tasks may be added or aborted mid-flight.
A cross-platform filename sanitizer and template renderer for output file naming. It handles Windows reserved names, control characters, path length limits, and template variables (e.g., {nickname}, {aweme_id}) with auto-truncation. The core functions are pure, stateless, and depend only on Python stdlib, making them directly reusable in any project that needs safe, configurable file naming.
A YAML configuration loader with environment variable overrides (prefixed DOUYIN_), alias normalization, schema validation, and logging of masked credentials. It merges default config, file config, and env overrides in a well-structured pipeline. The pattern—loading from YAML, overriding with env vars, validating, and merging—is generic and could be extracted for any Python application needing flexible, layered configuration.