$ show Colin-XKL/FeedCraft
FeedCraft is a self-hosted feed processing service that transforms raw sources (RSS, Atom, HTML, JSON, search results, inbox pushes) into customized RSS feeds through composable processing pipelines called 'crafts'. The system is a monolith Go backend with a Vue 3 admin dashboard, exposing REST APIs for feed ingestion, transformation (AI-powered translation, summarization, full-text extraction, filtering), and aggregation into 'topic feeds'. The architecture follows a clean separation of concerns: source fetching/parsing, craft processing (atomic operations and flows), engine runtime, and administrative controllers. External dependencies include SQLite/PostgreSQL, Redis, an OpenAI-compatible LLM, a browserless service for dynamic content, and optional embedding APIs for semantic filtering. The codebase demonstrates solid engineering with interfaces for sources, parsers, crafts, and caching, though it carries some migration complexity from legacy craft models to newer ones.
The LLM identified 5 files worth highlighting for reuse:
A generic priority-based dispatcher that manages a pool of workers with an urgent queue. Supports panic recovery, per-task timeout, and graceful shutdown. Reusable in any Go service needing prioritized concurrency control with urgent task preemption.
Provides CachedFunc and CachedFuncWithPreLog, wrapping expensive functions with Redis-backed caching and singleflight-based deduplication to prevent cache stampedes. Includes key generation and TTL management. A drop-in caching layer for any Go service.
Centralized LLM dispatcher that wraps multiple model providers with automatic retry, model fallback, priority queuing, and context-aware cancellation. Uses the priority dispatcher internally. A reusable LLM gateway pattern for applications needing resilient AI calls.
Implements zero-shot semantic filtering of feed items using cosine similarity between article embeddings and anchor vectors. Supports multiple embedding providers (OpenAI, Gemini, Ollama) with caching of anchor vectors. Extracted as a standalone craft processor, it can be reused in any content filtering pipeline requiring semantic relevance thresholds.
Provides TransformJSON for extracting values from nested JSON using a dot-separated path syntax (e.g., 'data.items') with array index support. Used in JSON-to-RSS conversion. This lightweight transformation utility is reusable for any Go service needing structured JSON field extraction without full jq dependency.