$ show xroche/httrack
HTTrack is a mature, production-grade website mirroring/offline browsing tool written in C. It recursively downloads websites to a local directory, rewriting links to enable offline browsing. The system includes a command-line interface (httrack), a web-based GUI (htsserver), a caching proxy (proxytrack), and a shared library (libhttrack). The core engine manages multi-threaded HTTP/HTTPS/FTP downloads, incremental updates via a zip-based cache, WARC/WACZ archival output, and robust link filtering. The architecture is monolithic but modularized into loosely coupled subsystems (caching, URL naming, HTML parsing, content decoding, robots.txt handling, etc.). Key entry points are in src/httrack.c, src/htsserver.c, and src/proxy/main.c. Data flows: user-provided URL seeds → DNS resolution → HTTP fetch → HTML parsing → link extraction → deduplication → resource download → local mirror tree or WARC output. External dependencies include zlib (required), OpenSSL (optional for WARC digests), and iconv (optional for charset conversion). The codebase is extensively tested with integration tests, fuzz harnesses, and self-tests; it also includes a standalone reusable test HTTP server and a WARC validator.
The LLM identified 5 files worth highlighting for reuse:
A header-only bump allocator that never moves or frees individual allocations. Provides O(1) allocation and bulk reset. Errors abort with a message. Fully self-contained and reusable in any C project needing a simple, fast allocator for scratch buffers or short-lived objects.
Provides macros for bounded string operations (strcpybuff, strcatbuff, slprintfbuff) that abort on overflow and support compile-time array size detection via HTS_SIZEOFARRAY. Also includes the htsbuff builder pattern for safe incremental string construction. Can be extracted as a standalone safe string library.
A pure-Python HTTP/HTTPS server (~2000 lines) used for offline, deterministic crawl testing. Supports cookies, redirects, content encodings, chunked transfer, partial downloads, MIME sniffing, robots.txt, sitemaps, and many edge cases. Stdlib only, no external dependencies. Reusable for testing any HTTP client or crawler.
A self-contained Python script that validates WARC files according to ISO 28500. Checks record syntax, mandatory fields, payload digest consistency, and structural integrity. Stdlib only; can be extracted and applied to any WARC archive for quality assurance.
A comprehensive charset/UTF-8 toolkit: strict UTF-8 validation, UCS-4 conversion, IDNA/punycode encoding/decoding, <meta charset> prescan, Windows codepage table lookup, and iconv fallback. Reusable for any C project that needs encoding-aware text processing without full ICU.