Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
This repository implements a persistent memory server for AI agents, designed to solve the problem of agent amnesia across sessions, devices, and team members. The production system is a Rust-based HTTP server (Axum) that exposes a RESTful API with 48+ endpoints, backed by LanceDB for vector storage and supporting both hosted and self-hosted deployments. It ingests conversational or document data via an LLM-powered pipeline that extracts atomic facts, filters noise, scores admission, and reconciles with existing memories using seven operations (CREATE, MERGE, SUPERSEDE, etc.). The system manages a full memory lifecycle with Weibull decay, tier-based promotion/demotion, and auto-forgetting, and provides a multi-stage retrieval pipeline that combines vector search, BM25, cross-encoder reranking, decay boosting, and MMR diversity. Plugins for Claude Code, OpenCode, OpenClaw, and MCP allow client-side integration, and a GitHub connector ingests commits, issues, and PRs as memories. The architecture is multi-tenant with personal, team, and organization spaces, each isolated in separate LanceDB directories, and supports versioned, provenance-tracked sharing with staleness detection.
Graphiti is a Python library and server framework for building temporal knowledge graphs that serve as persistent memory for AI agents. It ingests unstructured text episodes (messages, transcripts, JSON), uses LLMs to extract entities and relationships with bi-temporal validity windows, deduplicates against existing graph data, and provides hybrid search (semantic + BM25 + graph traversal) with configurable reranking. The codebase is structured around a clean driver abstraction (Neo4j, FalkorDB, Kuzu, Neptune) with per-provider operations classes, a prompt library supporting versioning and overrides, and a multi-provider LLM/embedder/cross-encoder client system. It is primarily an application library, though it includes evaluation infrastructure and research-grade deduplication algorithms (MinHash/LSH).
This is a production-grade long-term memory plugin for the OpenClaw AI agent platform, built as a TypeScript/ESM module. It provides a complete memory lifecycle: LLM-powered extraction from conversation transcripts (6-category classification with L0/L1/L2 layering), hybrid vector+BM25 retrieval with cross-encoder reranking, Weibull decay-based importance scoring, tier management (core/working/peripheral), and multi-scope isolation (global, agent, user, project). The plugin integrates deeply into OpenClaw via lifecycle hooks, agent tools, CLI commands, and a dreaming engine for periodic consolidation. It stores all data in LanceDB and supports multiple embedding providers, LLM backends, and OAuth flows. The codebase is large and complex, with extensive test coverage (100+ tests) and many edge-case handlers, but exhibits significant technical debt in a single massive index.ts file and multiple fallback/legacy migration paths. It is not a research project; it is a highly functional, actively maintained plugin with a clear engineering focus on correctness, resilience, and performance.