HTTrack is a mature, production-grade website mirroring/offline browsing tool written in C. It recursively downloads websites to a local directory, rewriting links to enable offline browsing. The system includes a command-line interface (httrack), a web-based GUI (htsserver), a caching proxy (proxytrack), and a shared library (libhttrack). The core engine manages multi-threaded HTTP/HTTPS/FTP downloads, incremental updates via a zip-based cache, WARC/WACZ archival output, and robust link filtering. The architecture is monolithic but modularized into loosely coupled subsystems (caching, URL naming, HTML parsing, content decoding, robots.txt handling, etc.). Key entry points are in src/httrack.c, src/htsserver.c, and src/proxy/main.c. Data flows: user-provided URL seeds → DNS resolution → HTTP fetch → HTML parsing → link extraction → deduplication → resource download → local mirror tree or WARC output. External dependencies include zlib (required), OpenSSL (optional for WARC digests), and iconv (optional for charset conversion). The codebase is extensively tested with integration tests, fuzz harnesses, and self-tests; it also includes a standalone reusable test HTTP server and a WARC validator.
ArchiveBox is a self-hosted internet archiving platform that captures web pages in multiple durable formats (HTML, PDF, screenshots, WARC, text, JSON) and stores them locally. It provides a CLI (rich-click), a Django web UI with admin interface, a REST API (Django Ninja), and background workers managed by supervisord. The system is event-driven via the `abx-dl` bus, with services projecting events to database models (Snapshot, ArchiveResult, Crawl, Process, Machine, Persona). Key entry points include the CLI commands (`add`, `extract`, `crawl`, `run`, `search`, `server`), the web UI at `/add/` and `/admin/`, and the REST API at `/api/v1/`. External dependencies include Python 3.13+, Django 6.1, SQLite/PostgreSQL, Chromium, wget, yt-dlp, and various Node.js tools (puppeteer, single-file-cli). The architecture is a Django monolith with a plugin-based extraction pipeline, supporting scheduled archiving, recursive crawling, Memento protocol, and cryptographic proof verification (TLSNotary, OpenTimestamps).