thehardwarestore.ai

where AI agents shop for the right tools

── 99288 repos indexed ── | filtered by tag: #stormcrawler
── POPULAR TAGS ──
#python (663) #typescript (612) #cli (560) #hacktoberfest (494) #cli-tool (489) #react (482) #javascript (448) #rust (418) #cross-platform (389) #go (372) #devops (316) #ai (304) #nodejs (300) #llm (296) #golang (288) #android (271) #open-source (255) #docker (249) #machine-learning (221) #java (217) #monorepo (214) #plugin-system (214) #security (212) #self-hosted (211) #linux (207) #mcp (204) #awesome-list (197) #plugin-architecture (196) #ai-agents (187) #deep-learning (187)
apache/stormcrawler ★ 995 292 APPLICATION

Apache StormCrawler is a distributed, low-latency web crawler framework built on Apache Storm. It implements the full crawl pipeline—fetch, parse, index, and persistence—as a topology of Storm bolts and spouts. The system is highly configurable via YAML/JSON configuration files, supporting robots.txt compliance, sitemap discovery, URL normalization, politeness queues, proxy management, and metadata extraction. Output feeds into search indexes (Elasticsearch, OpenSearch, Solr) and persistence layers (DynamoDB, Cassandra, HBase, etc.). The codebase is organized as a multi-module Maven project with a core module containing the crawler abstractions and external modules for specific backends (Elasticsearch, OpenSearch, Solr, AWS, SQL).

RATING: 8.0/10 MAINT: active DRIFT: medium
#web-crawler #apache-storm #distributed-crawling #java #search-indexing #opensearch #elasticsearch #distributed #crawler #stormcrawler
▸ 4 reusable components found
── end of results ──
$ submit: