$ show caolvchong-top/twitter_download
This repository provides a set of Python CLI tools for downloading media (images, videos, GIFs) and text from Twitter/X. It uses Twitter's internal GraphQL API (not the official API) and requires a valid user cookie for authentication. The primary entry point is `main.py`, which downloads media from a specified user's timeline, with options to include retweets, highlights, likes, and time range filtering. Additional scripts handle downloading by hashtag or advanced search (`tag_down.py`), downloading replies to a specific tweet or user (`reply_down.py`), downloading only text tweets (`text_down.py`), and downloading profile information (`profile_down.py`). The tool generates CSV logs and optional Markdown files, supports concurrent async downloads, and includes a caching mechanism to avoid re-downloading. It is a monolithic application with shared utility modules for CSV generation, Markdown generation, caching, and URL handling. The code is functional and well-documented in Chinese, but exhibits some code duplication and tight coupling to Twitter's API structure. It is actively maintained and adapted to Twitter's API changes.
The LLM identified 2 files worth highlighting for reuse:
A simple, self-contained class for caching downloaded media URLs using a pickle-backed set. It provides `add` and `is_present` methods to track already-downloaded items, preventing redundant downloads. The pattern is reusable in any project needing a lightweight, file-based cache for unique identifiers.
A utility module that wraps the `XClientTransaction` library to generate X-Client-Transaction-ID headers required by Twitter's API. It fetches the on-demand JavaScript file from Twitter's homepage to derive the transaction ID. This pattern demonstrates how to reverse-engineer client-side authentication tokens for unofficial API access.