$ show openclaw/slacrawl
slacrawl is a local-first Slack archiving CLI tool written in Go. It ingests messages, files, users, and channels from multiple sources—Slack API (bot/user tokens), Slack Desktop local cache (LevelDB + IndexedDB), MCP (Model Context Protocol) adapters, and external provider subprocesses—and stores them in a local SQLite database with full-text search (FTS5). The tool provides a rich command surface including sync, search, TUI browsing, analytics reports (digest, trends, quiet channels), retention purging, and Git-based archive sharing (publish/subscribe). It is designed for privacy-conscious users who want offline access to their Slack history without cloud dependency. The architecture follows a clean internal package separation: `cmd/slacrawl` is the entry point, `internal/cli` handles command routing via Kong, `internal/store` provides the SQLite persistence layer with schema migrations and batch writes, `internal/syncer` orchestrates multi-source synchronization, and `internal/slackapi`, `internal/slackdesktop`, `internal/slackmcp`, and `internal/provider` implement source-specific adapters. The codebase is well-tested with extensive Go tests, CI/CD via GitHub Actions, and cross-platform releases (Linux, macOS, Docker). While primarily an application, slacrawl exhibits data pipeline characteristics: it ingests raw Slack data, transforms it into a normalized relational schema with FTS indexing, and exposes analytical outputs (reports, digests). The embedding_jobs table hints at future ML integration. The Git archive sharing module is a novel pattern for distributed data replication. Overall, the codebase demonstrates strong engineering practices with reusable components like the generic MCP client, platform-specific file locking, and the external provider protocol.
The LLM identified 3 files worth highlighting for reuse:
A generic Model Context Protocol (MCP) client supporting HTTP and stdio transports. It handles tool listing, tool calling, and resource access with structured JSON-RPC messages. This component is fully self-contained (only depends on standard library and a JSON-RPC library) and can be extracted and reused in any Go project that needs to communicate with MCP servers, such as AI agent frameworks or custom tool integrations.
Implements file-based locking using flock(2) on Unix systems. Part of a pair with lock_windows.go (LockFileEx), this demonstrates clean OS-aware concurrency control for cache directories. The pattern is reusable for any Go application that needs cross-platform file locking with timeout and error handling, and can be extracted as a small utility package.
Defines a JSONL-based subprocess protocol for external data providers. It handles handshake (hello), workspace/channel/user/message ingestion, checkpointing, and error reporting over stdin/stdout. This component is a reusable pattern for integrating arbitrary data sources into a pipeline via a simple text protocol, and can be adapted for other ETL tools that need to stream structured data from external processes.