ArchiveBox is a self-hosted internet archiving platform that captures web pages in multiple durable formats (HTML, PDF, screenshots, WARC, text, JSON) and stores them locally. It provides a CLI (rich-click), a Django web UI with admin interface, a REST API (Django Ninja), and background workers managed by supervisord. The system is event-driven via the `abx-dl` bus, with services projecting events to database models (Snapshot, ArchiveResult, Crawl, Process, Machine, Persona). Key entry points include the CLI commands (`add`, `extract`, `crawl`, `run`, `search`, `server`), the web UI at `/add/` and `/admin/`, and the REST API at `/api/v1/`. External dependencies include Python 3.13+, Django 6.1, SQLite/PostgreSQL, Chromium, wget, yt-dlp, and various Node.js tools (puppeteer, single-file-cli). The architecture is a Django monolith with a plugin-based extraction pipeline, supporting scheduled archiving, recursive crawling, Memento protocol, and cryptographic proof verification (TLSNotary, OpenTimestamps).
`gau` (getallurls) is a high-performance Go CLI tool that aggregates known URLs for a given domain from multiple public web archives: Wayback Machine, CommonCrawl, AlienVault OTX, and URLScan. It is designed primarily for security researchers and penetration testers to enumerate URL endpoints, supporting concurrent fetches, filtering by status code/mime-type/date, blacklisting extensions, and deduplication of parameters. The architecture follows a clean provider interface pattern, with each data source implemented as a separate package satisfying a common `Provider` interface. The runner orchestrates a worker pool for concurrent processing, and output is handled via configurable JSON or plain text writers. The codebase is well-structured, testable in principle (though no tests are present in the visible tree), and the dependency graph is minimal and modern, relying on `fasthttp` for efficient HTTP requests and `viper` for configuration management.