$ show openclaw/photoscrawl
photoscrawl is a macOS CLI tool that crawls, archives, and analyzes Apple Photos libraries. It extracts metadata, faces, search indexes, and performs classification using local multimodal models (Ollama/OpenAI). The tool stores observations and evidence in a local SQLite database (photos.sqlite) rather than final truth, enabling later hypothesis generation for people, trips, and places. It provides subcommands for crawling, importing, classifying, searching, ranking, similarity, place backfill, and evaluation card generation. Architecture: The system is a Go CLI with a modular internal structure. It uses a crawlkit shared library for crawler mechanics, control, output, and store. The Apple Photos library is accessed via PhotoKit (cgo/Objective-C bridges) and direct SQLite snapshots. Data flows: snapshot Apple Photos database -> import metadata, faces, search indexes -> classify assets (metadata observations, then optional local model inference) -> store evidence in archive. Place backfill uses a subprocess bridge for reverse geocoding with rate limiting. Eval cards prepare images and run Ollama models for evaluation. Key entry points: cmd/photoscrawl/main.go (CLI), internal/archive/crawl.go (crawl logic), internal/archive/classify.go (classification queue), internal/place/backfill.go (batch geocoding), internal/evalcard/evalcard.go (model evaluation). External dependencies: Apple Photos (macOS), SQLite, Ollama/OpenAI (optional), crawlkit, Go standard library. The codebase demonstrates strong engineering practices with comprehensive tests, CI/CD, vulnerability scanning, and schema migrations. It is actively maintained with frequent dependency updates and a focus on privacy (all model calls local by default, no private data in repo).
The LLM identified 4 files worth highlighting for reuse:
Implements a robust SQLite database snapshotting pattern: verifies regular file, opens source, retries up to 5 attempts on concurrent changes, hashes the copy, runs quick_check, and returns a cleanup function. This is a self-contained utility that can be extracted for any project needing safe, consistent copies of live SQLite databases.
Implements a resumable classification queue with lease-based claiming to avoid double-processing. Claims rows with an owner and expiration; expired leases are reclaimed. This pattern is reusable for any batch processing system that needs fault-tolerant, distributed work distribution without a message broker.
Provides a batch reverse geocoding system using a subprocess bridge for isolation, with worker pool, rate limiting, and cooldown. The subprocess approach isolates external API calls and allows clean timeout handling. This pattern can be adapted for any batch enrichment task that requires external service calls with rate limits.
Implements database schema migrations with preflight validation: snapshotting for rollback journals, sidecar inspection, and hardlink rejection. The migration system ensures data integrity before applying changes. This pattern is reusable for any application needing safe, reversible schema migrations with validation.