$ show Marker-Inc-Korea/AutoRAG
AutoRAG is a production-grade, privacy-focused personal AI assistant and retrieval-augmented generation (RAG) agent. It provides a local-first search and answer engine that indexes federated datasources (Slack, Discord, KakaoTalk, GitHub, email, Obsidian, RSS, cloud drives, and more) via external CLI connectors, without migrating data. The system exposes a CLI, MCP server, TypeScript API, and a TUI (Pi interactive host), and powers a separate desktop Electron app. Its architecture is modular: a configurable agent (`AutoRAGAgent`) orchestrates multiple retrieval methods (BM25, semantic vectors, hybrid RRF, file-name search, web search), document parsing (PDF, DOCX, HWP, images via OCR), embedding (local Qwen3 model or external APIs), reranking (OpenRouter/Dazzi), and evidence judgment. A peer-to-peer layer (SimpleX) enables policy-controlled file sharing with injection and PII gates. The repository also contains a legacy Python codebase (`legacy/`) focused on automated RAG pipeline evaluation and QA dataset generation, which is in maintenance mode but still provides reusable metrics and pipeline components. This is a hybrid project where the dominant character is a modern TypeScript application, with significant data/research heritage in the legacy Python portion.
The LLM identified 5 files worth highlighting for reuse:
Provides a self-contained function `parseWithIsolatedWorker` that spawns a dedicated Worker thread for parsing untrusted files. Sets a hard memory limit, CPU time budget, and wall-clock timeout, terminating the worker if exceeded. Returns structured results with diagnostics. Reusable in any application needing safe, bounded document processing without risking the main process.
Implements `ResultMerger` that takes retrieval results from multiple methods (BM25, semantic, reranked) and fuses them into a single ranked list. Uses min-max normalization scaled by method weight, then sums scores across methods with a corroboration bonus for duplicate evidence IDs. Includes deduplication and diagnostic tracking. Self-contained and reusable for any multi-strategy fusion pipeline.
Manages local download cache for embedding model runtimes. `downloadAndExtract` downloads a URL, verifies SHA-256 hash, atomically extracts archives, and writes a provisioned marker. `resolveCachedArchive` and `cachePath` provide lookup. Handles retries and stale cache cleanup. Reusable pattern for any tool that needs reliable, integrity-checked asset caching.
Implements `PIIRedactor` that scans text for personally identifiable information (email, phone, credit card, SSN, IP, etc.) using regex patterns, then applies stable pseudonyms or redaction markers. Includes optional NER-based detection via a provider. Maintains a deterministic mapping per session to preserve referential consistency. Standalone utility for any system needing PII sanitization.
Provides `fileLock` and `fileLockWithRetry` functions using directory-based locking (mkdir which is atomic on POSIX and Windows). Includes stale lock detection via PID and timestamp, configurable retry with backoff, and cleanup on SIGINT. Windows-specific retry for transient access-denied errors. Reusable for any application needing simple, cross-process file locks.