HTTrack is a mature, production-grade website mirroring/offline browsing tool written in C. It recursively downloads websites to a local directory, rewriting links to enable offline browsing. The system includes a command-line interface (httrack), a web-based GUI (htsserver), a caching proxy (proxytrack), and a shared library (libhttrack). The core engine manages multi-threaded HTTP/HTTPS/FTP downloads, incremental updates via a zip-based cache, WARC/WACZ archival output, and robust link filtering. The architecture is monolithic but modularized into loosely coupled subsystems (caching, URL naming, HTML parsing, content decoding, robots.txt handling, etc.). Key entry points are in src/httrack.c, src/htsserver.c, and src/proxy/main.c. Data flows: user-provided URL seeds → DNS resolution → HTTP fetch → HTML parsing → link extraction → deduplication → resource download → local mirror tree or WARC output. External dependencies include zlib (required), OpenSSL (optional for WARC digests), and iconv (optional for charset conversion). The codebase is extensively tested with integration tests, fuzz harnesses, and self-tests; it also includes a standalone reusable test HTTP server and a WARC validator.
ArchiveBox is a self-hosted internet archiving platform that captures web pages in multiple durable formats (HTML, PDF, screenshots, WARC, text, JSON) and stores them locally. It provides a CLI (rich-click), a Django web UI with admin interface, a REST API (Django Ninja), and background workers managed by supervisord. The system is event-driven via the `abx-dl` bus, with services projecting events to database models (Snapshot, ArchiveResult, Crawl, Process, Machine, Persona). Key entry points include the CLI commands (`add`, `extract`, `crawl`, `run`, `search`, `server`), the web UI at `/add/` and `/admin/`, and the REST API at `/api/v1/`. External dependencies include Python 3.13+, Django 6.1, SQLite/PostgreSQL, Chromium, wget, yt-dlp, and various Node.js tools (puppeteer, single-file-cli). The architecture is a Django monolith with a plugin-based extraction pipeline, supporting scheduled archiving, recursive crawling, Memento protocol, and cryptographic proof verification (TLSNotary, OpenTimestamps).
SingleFileZ is a browser extension (Chrome/Firefox/Edge) and CLI tool that saves complete web pages as self-extracting HTML/ZIP polyglot files. The core output format embeds a ZIP archive containing all page resources (HTML, CSS, images, fonts, frames) inside an HTML wrapper with a JavaScript bootstrap that extracts and displays the page without external dependencies. The extension offers extensive configuration including multiple profiles, URL-based auto-save rules, cloud uploads (Google Drive, Dropbox, GitHub, WebDAV), an annotation editor for notes/highlights/cut operations, batch saving, and file name templating. Architecturally, the project is a multi-browser extension with background scripts orchestrating page capture, content scripts handling in-page UI and resource collection, and a task queue for managing parallel saves. It relies heavily on message passing between contexts and includes custom infrastructure: a binary serialization format (YABSON) for efficient data transfer, a full ZIP read/write library with support for encryption and streaming, and a fork of Mozilla's Readability.js for article extraction. The CLI component uses Docker to run headlessly, delegating to the `single-filez-cli` npm package. The codebase is large and feature-rich but shows signs of maintainability strain: core modules are distributed as minified build artifacts in `lib/`, the README states SingleFileZ has been merged into its parent project SingleFile (suggesting this repo is superseded), and it relies on Manifest v2 which is being phased out by browsers. The custom polyfills and browser-specific workarounds add complexity. Despite this, several components (YABSON serialization, the ZIP library, cloud upload abstractions) are well-isolated and could be reused independently with minimal modification.