$ show thisisparker/xword-dl
xword-dl is a command-line tool that downloads crossword puzzles from over 30 online outlets and saves them in the standard .puz format. The codebase implements a plugin-based architecture where each outlet (e.g., New York Times, Guardian, Los Angeles Times) has its own downloader subclassing a base class. The main entry point (xword_dl.py) discovers plugins via package walking, parses command-line arguments, and dispatches to the appropriate downloader. The download flow consists of locating the puzzle URL (by latest or by date), fetching solver data, parsing the outlet-specific format (JSON, XML, or obfuscated payload) into a unified Puzzle object, sanitizing content for the .puz format, and saving the file. Notable technical challenges handled include deobfuscating AmuseLabs' proprietary encoding using a brute-force BFS key search, authenticating with NYT's subscription API, and translating diverse grid/clue representations into a standardized binary format. The project is actively maintained with pinned dependencies, GitHub Actions CI, and a well-documented plugin pattern. While it lacks unit tests and some downloaders are disabled due to anti-scraping measures, the core architecture is clean and the deobfuscation module is a standout reusable asset.
The LLM identified 2 files worth highlighting for reuse:
Contains the deobfuscate_rawc function that reverse-engineers AmuseLabs' obfuscated crossword data. It uses a BFS search over possible chunk-reversal keys, validates candidate keys by checking UTF-8 validity of decoded base64 chunks, and returns the parsed JSON. This logic is self-contained, depends only on standard libraries (base64, json, collections), and could be extracted and reused in any project that needs to decode AmuseLabs rawc strings without modification.
Provides sanitize_for_puzfile and cleanup functions that prepare crossword data for the .puz format. They handle HTML-to-text conversion (via html2text), emoji demojization, unidecode for Latin-1 compatibility, and cleaning of special characters. These functions are parametrized (preserve_html flag) and operate purely on Puzzle objects and strings, making them reusable in any project that processes crossword puzzle data for storage in the .puz standard.