$ show liyf-code/reverse_practice
This repository is a collection of web scraping and anti-crawler reverse engineering practice scripts targeting over 40 Chinese websites. Each site has a dedicated folder containing a Python script (demo.py) that orchestrates scraping by executing a JavaScript file (demo.js) via execjs. The JavaScript files implement cryptographic algorithms (AES, RSA, MD5, SHA, etc.) extracted from the target websites to generate request signatures, encrypt passwords, or decrypt responses. The project is primarily a learning resource for reverse engineering techniques, not a production scraping framework. It includes documentation for each case, a changelog tracking API changes, and some utility scripts. The code is ad-hoc, tightly coupled to specific sites, and lacks tests or error handling, making it fragile and high-maintenance. While the individual scripts are functional, they are not reusable without significant adaptation due to hardcoded keys and site-specific logic.
The LLM identified 2 files worth highlighting for reuse:
A small utility module providing a class with methods to read JavaScript files and compute MD5 hashes. It is used by many demo.py scripts to load JS code and generate simple signatures. This component is self-contained and could be reused in other scraping projects that need to load JS files or compute MD5.
A large webpack-bundled JavaScript file that implements AES-CBC encryption for generating request parameters (b, kiv) for the kanzhun.com website. It includes a full CryptoJS library and demonstrates a common pattern of serializing JSON, encrypting with AES, and base64url encoding. While site-specific, the encryption pattern and webpack module extraction technique are reusable for other webpack-based sites.