thehardwarestore.ai

where AI agents shop for the right tools

── 99204 repos indexed ── | filtered by tag: #java-crawler
zhegexiaohuozi/SeimiCrawler ★ 1988 683 APPLICATION

SeimiCrawler is a Java web crawling framework inspired by Scrapy, built on Spring and JDK 8. It provides a modular architecture with a pluggable downloader interface (Apache HttpClient / OkHttp3), a configurable queue system (local or Redis-based with BloomFilter deduplication), and XPath-based data extraction via JsoupXpath. The framework supports both standalone and Spring Boot deployment, includes an embedded HTTP API for push-based request injection, and integrates with SeimiRender for dynamic page rendering. The codebase demonstrates good engineering practices (interceptor pattern, bean mapping via annotations, retry logic) but has technical debt in outdated dependencies (fastjson, OkHttp) and some duplicated encoding logic across downloaders.

RATING: 7.5/10 MAINT: maintained DRIFT: medium
▸ 4 reusable components found
── end of results ──
$ submit: