$ show lc/gau
`gau` (getallurls) is a high-performance Go CLI tool that aggregates known URLs for a given domain from multiple public web archives: Wayback Machine, CommonCrawl, AlienVault OTX, and URLScan. It is designed primarily for security researchers and penetration testers to enumerate URL endpoints, supporting concurrent fetches, filtering by status code/mime-type/date, blacklisting extensions, and deduplication of parameters. The architecture follows a clean provider interface pattern, with each data source implemented as a separate package satisfying a common `Provider` interface. The runner orchestrates a worker pool for concurrent processing, and output is handled via configurable JSON or plain text writers. The codebase is well-structured, testable in principle (though no tests are present in the visible tree), and the dependency graph is minimal and modern, relying on `fasthttp` for efficient HTTP requests and `viper` for configuration management.
The LLM identified 3 files worth highlighting for reuse:
Encapsulates date and status/mime-type filter parameters into query strings tailored to Wayback Machine and CommonCrawl API conventions. This is a clean builder pattern that could be reused in any project that needs to generate archive filter parameters from a set of user-defined rules.
A reusable HTTP client wrapper around fasthttp that implements automatic retries on non-400 errors, configurable timeouts, random User-Agent rotation, and custom header injection. It is decoupled from the rest of gau and could be extracted as a standalone library for any Go application requiring an efficient, resilient HTTP client.
Provides output functions for writing URL results to an io.Writer, supporting plain text and JSON formats, optional blacklisting by file extension, and deduplication of parameterized URLs. The design is generic and can be reused for any streaming output pipeline that requires filtering and deduplication.