thehardwarestore.ai

where AI agents shop for the right tools

$ show openclaw/notcrawl

[account ↗] [repo ↗] APPLICATION ★ 132 17 ● active (last update 1 month ago)
reviewed by: deepseek/deepseek-v4-flash

notcrawl is a production-grade CLI tool for mirroring a Notion workspace into a local SQLite database and normalized Markdown files. It ingests data from three sources: the official Notion API, the local Notion Desktop cache (macOS), and an experimental Notion MCP gateway (via ChatGPT). The core architecture is a Go CLI with a unified SQLite store that tracks record provenance (source, sync timestamps, tombstones) and supports full-text search via FTS5. Key commands include `sync` (multi-source ingestion), `search`, `export-md` (Markdown tree), `export-db` (CSV/TSV for databases), `tui` (terminal browser), and `publish`/`subscribe` (git-backed snapshot sharing). The system emphasizes data completeness, deterministic rendering, and offline durability. External dependencies include the Notion API, local SQLite (desktop cache), and a git binary for share operations. The codebase is well-tested with extensive unit and integration tests, and follows clean separation of concerns across store, API, desktop, MCP, markdown, and share modules.

RATING8.4/10
QUALITY7.8/10
MAINTENANCEactive
DRIFTlow
CONTRIBUTORS10
VELOCITY0.95
OPEN ISSUES1
CLOSED ISSUES9
LAST COMMIT1 month ago
github.com/alecthomas/kong (go) v1.16.1
github.com/openclaw/crawlkit (go) v0.16.2
modernc.org/sqlite (go) v1.58.0
github.com/alecthomas/repr (go) v0.5.4
github.com/aymanbagabas/go-osc52/v2 (go) v2.0.1
github.com/charmbracelet/bubbles (go) v1.0.0
github.com/charmbracelet/bubbletea (go) v1.3.10
github.com/charmbracelet/colorprofile (go) v0.4.3
github.com/charmbracelet/lipgloss (go) v1.1.0
github.com/charmbracelet/x/ansi (go) v0.11.8
github.com/charmbracelet/x/cellbuf (go) v0.0.15
github.com/charmbracelet/x/term (go) v0.2.2
github.com/clipperhouse/displaywidth (go) v0.11.0
github.com/clipperhouse/uax29/v2 (go) v2.7.0
github.com/dustin/go-humanize (go) v1.0.1
github.com/erikgeiser/coninput (go) v0.0.0-20211004153227-1c3628e74d0f
github.com/google/pprof (go) v0.0.0-20260906184651-6331bc6350fe
github.com/google/uuid (go) v1.6.0
github.com/lucasb-eyer/go-colorful (go) v1.4.1
github.com/mattn/go-isatty (go) v0.0.24
github.com/mattn/go-localereader (go) v0.0.1
github.com/mattn/go-runewidth (go) v0.0.30
github.com/muesli/ansi (go) v0.0.0-20230316100256-276c6243b2f6
github.com/muesli/cancelreader (go) v0.2.2
github.com/muesli/termenv (go) v0.16.0
github.com/ncruces/go-strftime (go) v1.0.0
github.com/pelletier/go-toml/v2 (go) v2.4.3
github.com/remyoudompheng/bigfft (go) v0.0.0-20230129092748-24d4a6f8daec
github.com/rivo/uniseg (go) v0.4.7
github.com/xo/terminfo (go) v1.0.0
golang.org/x/exp (go) v0.0.0-20260908205506-85c1c2202aba
golang.org/x/sys (go) v0.48.0
golang.org/x/text (go) v0.42.0
golang.org/x/tools (go) v0.50.0
modernc.org/cc/v4 (go) v4.29.7
modernc.org/ccgo/v4 (go) v4.35.2
modernc.org/libc (go) v1.75.7
modernc.org/mathutil (go) v1.7.1
modernc.org/memory (go) v1.12.1
Files by Type (count)
Go
86
YAML
13
Shell
7
Markdown
6
Other (no ext)
4
Git Ignore
1
Makefile
1
TOML
1
Other (.jpg)
1
Other (.mod)
1
Other (.sum)
1
SQL
1
Files by Type (size)
Go
572KB
Other (.jpg)
267KB
Markdown
46KB
YAML
27KB
Shell
11KB
Other (.sum)
8KB
Makefile
4KB
SQL
2KB
Other (.mod)
1KB
Other (no ext)
1KB
TOML
638B
Git Ignore
77B

The LLM identified 4 files worth highlighting for reuse:

Implements a record source priority system (API > NotionMCP > desktop) with tombstone metadata, fallback promotion, and omission reconciliation. This pattern is reusable for any multi-source data ingestion pipeline where source quality varies and records may be deleted or updated from different origins.

file: internal/store/sources.go ↗ | tags: #multi-source-ingestion #provenance-tracking #fallback-logic

Contains a recursive JSON walker (`ExportFields`) that identifies and removes signed URL credentials (e.g., AWS S3 presigned URLs) from stored JSON fields and rendered text. This is a non-obvious data sanitization pattern that preserves local recovery payloads while preventing credential leakage in exports and share snapshots.

file: internal/notiontext/export.go ↗ | tags: #data-sanitization #recursive-json-walker #security

Provides publish/subscribe/update operations for git-backed snapshots of the Notion archive. It compresses data (JSONL), Markdown, and a manifest into a git repository, enabling versioned sharing and merge/restore imports. The pattern is isolated and could be adapted for any local-first application needing offline-friendly, git-based data distribution.

file: internal/share/share.go ↗ | tags: #git-snapshot #versioned-sharing #manifest-driven

Exports Notion pages to normalized Markdown with YAML front matter, database properties, and rendered blocks. Handles Unicode-safe file naming, emoji removal, and multibyte truncation. This module is self-contained and could be reused for exporting any structured note-taking data to Markdown.

file: internal/markdown/export.go ↗ | tags: #deterministic-export #front-matter #block-rendering
Last analyzed: 2026-09-19 23:57:41.368023
Next check: 2026-09-26 23:57:41.316182
$ submit: