$ show zhegexiaohuozi/SeimiCrawler
SeimiCrawler is a Java web crawling framework inspired by Scrapy, built on Spring and JDK 8. It provides a modular architecture with a pluggable downloader interface (Apache HttpClient / OkHttp3), a configurable queue system (local or Redis-based with BloomFilter deduplication), and XPath-based data extraction via JsoupXpath. The framework supports both standalone and Spring Boot deployment, includes an embedded HTTP API for push-based request injection, and integrates with SeimiRender for dynamic page rendering. The codebase demonstrates good engineering practices (interceptor pattern, bean mapping via annotations, retry logic) but has technical debt in outdated dependencies (fastjson, OkHttp) and some duplicated encoding logic across downloaders.
The LLM identified 4 files worth highlighting for reuse:
Interface defining a pluggable queue for crawler requests. Implementations include DefaultLocalQueue (in-memory blocking queue) and DefaultRedisQueue (Redisson-based with BloomFilter). This pattern enables easy switching between single-machine and distributed processing, and can be reused in any crawler framework needing a decoupled request queue.
Interface for HTTP downloading, with implementations for Apache HttpClient and OkHttp3. This abstraction allows per-request selection of the HTTP client and enables custom downloaders. The pattern is cleanly isolated and can be extracted to decouple HTTP logic from any application.
Fluent builder for constructing crawl requests with support for URL, method, headers, cookies, JSON body, retry count, duplicate filtering, and SeimiRender options. The builder pattern with lambda callback support is a reusable design for request construction in any crawling or HTTP client library.
Annotation for mapping XPath expressions to Java bean fields. Used by Response.render() to automatically populate POJOs from HTML documents. This pattern of annotation-driven extraction is reusable for any structured data parsing from HTML, not just crawling.