Self-contained parsers for robots.txt directives and sitemap discovery (including strict mode and extension parsing). These implement standard web-crawling protocols (RFC 9309 for robots.txt, sitemaps.org for sitemaps) and are not coupled to Storm. An engineer could extract them into any crawling, SEO, or link-analysis project with minimal modification.

URL normalization and filtering utilities that implement canonicalization rules, deduplication, and cross-domain filtering. These are standard web-crawling primitives that solve the recurring problem of URL canonicalization in any crawling or data-deduplication pipeline. The logic is isolated from the Storm topology and could be reused in a standalone library.

WARC (Web ARChive) format handling for recording crawl data. WARC is an ISO standard (ISO 28500) for web archiving, and a clean implementation of its read/write logic is reusable across web archiving, digital preservation, and research crawling projects. The implementation demonstrates careful handling of the format's edge cases, though one feature (request ID storage) is currently deactivated pending issue #721.

Abstract base class implementing the template method pattern for status persistence in a crawl pipeline. Subclasses provide backend-specific implementations (Elasticsearch, OpenSearch, Solr, SQL, etc.) while the base class handles the common workflow of receiving status updates, batching, and delegating to the backend. This demonstrates a clean pluggable-persistence architecture that could be adapted for other distributed stream-processing frameworks.