$ show openclaw/notcrawl
notcrawl is a production-grade CLI tool for mirroring a Notion workspace into a local SQLite database and normalized Markdown files. It ingests data from three sources: the official Notion API, the local Notion Desktop cache (macOS), and an experimental Notion MCP gateway (via ChatGPT). The core architecture is a Go CLI with a unified SQLite store that tracks record provenance (source, sync timestamps, tombstones) and supports full-text search via FTS5. Key commands include `sync` (multi-source ingestion), `search`, `export-md` (Markdown tree), `export-db` (CSV/TSV for databases), `tui` (terminal browser), and `publish`/`subscribe` (git-backed snapshot sharing). The system emphasizes data completeness, deterministic rendering, and offline durability. External dependencies include the Notion API, local SQLite (desktop cache), and a git binary for share operations. The codebase is well-tested with extensive unit and integration tests, and follows clean separation of concerns across store, API, desktop, MCP, markdown, and share modules.
The LLM identified 4 files worth highlighting for reuse:
Implements a record source priority system (API > NotionMCP > desktop) with tombstone metadata, fallback promotion, and omission reconciliation. This pattern is reusable for any multi-source data ingestion pipeline where source quality varies and records may be deleted or updated from different origins.
Contains a recursive JSON walker (`ExportFields`) that identifies and removes signed URL credentials (e.g., AWS S3 presigned URLs) from stored JSON fields and rendered text. This is a non-obvious data sanitization pattern that preserves local recovery payloads while preventing credential leakage in exports and share snapshots.
Provides publish/subscribe/update operations for git-backed snapshots of the Notion archive. It compresses data (JSONL), Markdown, and a manifest into a git repository, enabling versioned sharing and merge/restore imports. The pattern is isolated and could be adapted for any local-first application needing offline-friendly, git-based data distribution.
Exports Notion pages to normalized Markdown with YAML front matter, database properties, and rendered blocks. Handles Unicode-safe file naming, emoji removal, and multibyte truncation. This module is self-contained and could be reused for exporting any structured note-taking data to Markdown.