$ show openclaw/crawlkit
Crawlkit is a shared Go library and CLI tool (crawlctl) for building local-first crawler archives. It provides provider-neutral building blocks including SQLite stores with FTS5, snapshot/export to gzipped JSONL shards, Git-based mirroring with immutable tags, encrypted backup with age/gzip and SHA-256 deduplication, sync state management with legacy adapters, vector search (exact cosine similarity and optional turbovec Python bridge), embedding providers (OpenAI, Ollama, llama.cpp), a terminal UI (Bubble Tea) for browsing archive rows, and a remote archive client for Cloudflare Workers. The CLI discovers installed crawl apps (e.g., gitcrawl, discrawl) and runs configured refresh jobs under a single-process lock. The architecture is modular and composable, with clear ownership boundaries documented in docs/boundary.md. The repository also includes agent skills (autoreview, crabbox) for automated code review and remote validation, reflecting heavy agent-driven development. It is not a generic web crawler but a mechanics layer that makes each crawl app smaller and more uniform.
The LLM identified 5 files worth highlighting for reuse:
Thread-safe progress tracker with configurable minimum delta and log interval. Supports CI-safe output by detecting non-TTY environments. Demonstrates careful synchronization and defensive programming for logging in concurrent contexts.
Format resolution utility that parses output format flags (text, json, log) and provides a UsageError type for validation errors. Clean separation of concerns and reusable across any CLI tool needing structured output.
Cross-platform runtime path resolution using XDG Base Directory on Linux/macOS with environment overrides and legacy fallback. Handles platform-specific quirks (e.g., Homebrew on Apple Silicon) and provides a clean DefaultPaths() API.
Atomic encrypted snapshot writer using age/gzip with SHA-256 deduplication. Writes encrypted shards under a data/ directory with a manifest. Restore verifies hashes and rejects unsafe paths. Demonstrates careful atomic file operations and cryptographic integrity.
Git repository mirroring with branch synchronization, rebase with local commit preservation, retargeting of unpublished tags, and atomic push. Handles non-fast-forward scenarios gracefully. Demonstrates deep understanding of Git internals and safe state management.