Miniflux is a self-hosted, minimalist RSS/Atom feed reader implemented as a monolithic Go web application. It relies exclusively on PostgreSQL for persistence and serves a server-rendered web UI, a comprehensive REST API, and compatibility endpoints for the Fever and Google Reader APIs. The application is designed for low resource consumption and easy deployment, evidenced by its single-binary nature and extensive packaging (Docker, RPM, Deb, systemd). The core strength of the codebase lies in its feed ingestion pipeline, which demonstrates exceptional defensive programming against the chaotic and non-standardized world of web feeds. This pipeline includes robust format detection (RSS, Atom, RDF, JSON Feed), character encoding detection for legacy charsets, a port of the Mozilla Arc90 readability algorithm for content extraction, a highly restrictive HTML sanitizer with SSRF protection, and a DSL-based content rewrite engine. The system supports multi-user authentication (local, OAuth2, WebAuthn), extensive third-party integrations (Discord, Slack, Telegram, Pinboard, etc.), and background job processing for feed refreshes. The codebase exhibits high craftsmanship with clean abstractions, consistent builder patterns, and thorough testing (unit, integration, fuzz). Several components, such as the SSRF-safe HTTP client builder, the readability extraction algorithm, the i18n pluralization engine, and the HTML sanitizer, are isolated, well-documented, and could be extracted for use in other Go projects. The project is actively maintained with modern Go features, zero Dependabot alerts, and a strong CI/CD pipeline. The main maintenance burden is the sprawl of third-party integrations and the monolithic JavaScript frontend, but the core Go server code is exceptionally well-engineered.
Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
Defuddle is a content extraction library that converts web pages into clean, readable Markdown with structured metadata (title, author, site, published date). It is the engine behind the Obsidian Web Clipper browser extension and also runs as a standalone CLI tool, an npm package, and a hosted API service on Cloudflare Workers. The architecture follows a modular pipeline: fetch HTML, detect site-specific extractors, score and select main content, remove clutter (nav, ads, sidebars), standardize complex elements (footnotes, math, code blocks, images), and convert to Markdown via Turndown. The library is designed as a more forgiving alternative to Mozilla's Readability, with support for mobile-style heuristics, schema.org metadata extraction, and extensive site-specific extractors for platforms like YouTube, Reddit, Twitter/X, ChatGPT, GitHub, and Gmail. The codebase is written in TypeScript and produces three bundles: core (browser, no extra dependencies), full (adds MathML-to-LaTeX conversion), and node (accepts any DOM implementation). The CLI uses Commander and supports file, URL, and stdin input with options for Markdown, JSON, frontmatter, and debug output. The hosted API runs on Cloudflare Workers with Stripe-based billing, rate limiting, and Durable Objects for stateful API key management. The test suite is exceptionally comprehensive, with hundreds of HTML fixtures and expected Markdown outputs covering edge cases from dozens of real-world websites, CMS patterns, and syntax highlighters. Key external dependencies include Turndown (Markdown conversion), linkedom (DOM parsing in Node.js), mathml-to-latex and Temml (math rendering), and various third-party APIs for site-specific extractors (YouTube InnerTube, FxTwitter, Reddit Atom feeds, Bilibili). The project maintains strict security practices: no innerHTML assignments outside a single utility function, URL sanitization, and XSS prevention through template-based HTML parsing. The build system uses webpack with aliases for math module switching and enforces bundle size budgets via CI scripts.
This is a Deno/TypeScript library for extracting structured article data (title, content, author, date, image, etc.) from any URL or HTML string. It uses Mozilla's Readability for the core extraction, supplemented by custom metadata extraction from HTML meta tags, JSON-LD schemas, and link elements. The library provides a transformation system that allows per-site pre/post processing of HTML to improve extraction quality, and a flexible HTML sanitizer to control output tags and attributes. The architecture is modular, with clear separation of concerns: retrieval, metadata extraction, URL normalization, readability extraction, sanitization, and transformation. Entry points are `extract()` and `extractFromHtml()`, with optional custom fetch functions and parser options. The codebase is well-tested, actively maintained, and demonstrates solid engineering practices with minimal dependencies.
Open Notebook is a self-hosted, AI-powered research assistant and knowledge management application. It allows users to collect sources (web pages, uploaded documents, text snippets), extract structured insights via LLM-driven transformations, organize content into notebooks, perform hybrid text/vector search, and generate AI podcasts with multi-speaker support using TTS models. The backend is built on Python with FastAPI and LangGraph for orchestration, SurrealDB for persistence (with a custom migration system), and the Esperanto library for multi-provider AI integration (OpenAI, Anthropic, Google, Ollama, and many others). The frontend is a modern Next.js application with React Query, i18n support for 14 locales, and a component library based on Radix UI/shadcn. The architecture follows a monolith-with-background-workers pattern: the FastAPI process handles both synchronous API requests and, via the surreal-commands library, asynchronous background jobs for content processing, embedding generation, and podcast generation. The codebase is well-structured with clear separation between API routers, domain models, utility modules, and frontend components. It exhibits strong security awareness—includes SSRF protection via DNS pinning, path containment for file uploads, Fernet encryption for stored credentials, and input validation throughout. The test suite is extensive (>100 test files) covering API endpoints, domain logic, error handling, and edge cases. There are some maintenance risks related to tight coupling with SurrealDB and the externally-maintained Esperanto/content-core/podcast-creator libraries, but overall the codebase is actively maintained and of professional quality.
This is an Obsidian plugin that integrates large language models (LLMs) directly into the note-taking environment. It provides a template-driven text generation system using Handlebars with custom async helpers, supports multiple LLM providers (OpenAI, Anthropic, Google Gemini, Mistral, Ollama, Perplexity, HuggingFace, Together AI, MiniMax, Azure, and custom endpoints), and offers content extraction from PDF, audio, images, YouTube, RSS, and web pages. The plugin includes a community template marketplace, auto-suggest, an overlay toolbar, a playground for testing templates, and extensive Obsidian integration (commands, views, settings, CodeMirror extensions). Architecturally, it is a client-side monolithic plugin that communicates directly with external LLM APIs and the npm registry for package management. The codebase is large (300+ files) and well-organized into providers, extractors, UI components, and services, but it shows some drift due to hard-coded model lists and deprecated provider support (e.g., Google PaLM). The async Handlebars helpers and generic caching utility are highlight components that could be extracted for reuse in other projects.
MMSearch is a framework for evaluating large multimodal models (LMMs) as multi-modal search engines. It provides a pipeline (MMSearch-Engine) that enables any LMM to perform multimodal search by generating queries, retrieving web content, reranking results, and summarizing answers. The repository includes evaluation scripts for four tasks (end2end, requery, rerank, summarization) and a demo for running the engine on custom queries. The system relies on external dependencies like DuckDuckGo search, Playwright for web scraping, and Hugging Face datasets. The architecture is modular with separate components for model loading, prompt templating, web retrieval, content processing, and scoring.