Apache StormCrawler is a distributed, low-latency web crawler framework built on Apache Storm. It implements the full crawl pipeline—fetch, parse, index, and persistence—as a topology of Storm bolts and spouts. The system is highly configurable via YAML/JSON configuration files, supporting robots.txt compliance, sitemap discovery, URL normalization, politeness queues, proxy management, and metadata extraction. Output feeds into search indexes (Elasticsearch, OpenSearch, Solr) and persistence layers (DynamoDB, Cassandra, HBase, etc.). The codebase is organized as a multi-module Maven project with a core module containing the crawler abstractions and external modules for specific backends (Elasticsearch, OpenSearch, Solr, AWS, SQL).
SeimiCrawler is a Java web crawling framework inspired by Scrapy, built on Spring and JDK 8. It provides a modular architecture with a pluggable downloader interface (Apache HttpClient / OkHttp3), a configurable queue system (local or Redis-based with BloomFilter deduplication), and XPath-based data extraction via JsoupXpath. The framework supports both standalone and Spring Boot deployment, includes an embedded HTTP API for push-based request injection, and integrates with SeimiRender for dynamic page rendering. The codebase demonstrates good engineering practices (interceptor pattern, bean mapping via annotations, retry logic) but has technical debt in outdated dependencies (fastjson, OkHttp) and some duplicated encoding logic across downloaders.