$ show idosal/git-mcp
GitMCP is a remote Model Context Protocol (MCP) server running on Cloudflare Workers that turns any public GitHub repository (or GitHub Pages site) into live, searchable documentation for AI tools. AI clients like Claude Desktop, Cursor, and MCP Inspector connect to gitmcp.io/{owner}/{repo} over SSE with OAuth and receive repo-specific MCP tools: fetch_<repo>_documentation, search_<repo>_documentation, search_<repo>_code, and fetch_url_content. The ingestion pipeline fetches llms.txt first (falling back to README.md, then landing-page HTML converted via html-to-md), chunks it with markdown-aware chunkers (heading-context-preserving, code-block and list aware), and indexes it into Cloudflare Vectorize using a dependency-free deterministic embedder built from hashing plus term-frequency/keyword extraction, with a hybrid ranking that blends normalized vector similarity (60%) with keyword/proximity/heading heuristics (40%). Robots.txt is parsed, cached, and respected before external fetches - an uncommon discipline for doc scrapers. The codebase also ships a full web chat application (React Router v7, Tailwind, shadcn/ui, AI SDK) with model selection across OpenAI/Anthropic/Groq/XAI, MCP server management, and a streaming chat API endpoint. Architecturally, the MCP server itself is implemented as a Cloudflare Durable Object (MyMCP), with KV caching (TTL with jitter to prevent cache stampedes), R2 storage for pre-generated docs, a queue for background ingestion, Vectorize for semantic search, Analytics Engine for usage metrics, and GitHub API integration with rate-limit tracking and retry/backoff. A notable design pattern is the RepoHandler strategy interface: per-repo families (three.js, react-router) provide customized tool sets, doc-fetch strategies, and search behavior while the generic handler covers all other repos, including a static mapping of library names to their canonical doc repos. Testing is solid: Vitest unit tests cover tool registration, repo handlers, URL parsing, caching, and three.js utilities (with snapshots), and Playwright e2e tests run against a locally-provisioned R2 environment; CI runs both on push/PR. Weaknesses and drift: the embedding layer is explicitly a prototype-grade stand-in ('In a production environment, you would use a proper embedding service like OpenAI') and will produce weak semantic results; badge.ts contains ~96k tokens of hardcoded escaped SVG logo paths (extreme maintenance fragility); several hardcoded host URLs (git-mcp.idosalomon.workers.dev, CHAT_API_URL) baked into source; a generated-type mismatch where Vectorize metadata filters use $gt at runtime but generated types only declare $eq/$ne; vector re-ingestion only deletes the first 100 stale chunks (repos with more chunks accumulate orphans); and the frontend uses React Router v7's unstable_viteEnvironmentApi. These are prototype-velocity marks rather than architectural rot - the core strategy/caching/testing structure is genuinely well-built.
The LLM identified 5 files worth highlighting for reuse:
A self-contained document ingestion and retrieval pipeline for Cloudflare Vectorize: markdown-aware chunkers (llms.txt structured docs, README, generic text with heading/code-block/list preservation), a dependency-free deterministic embedding built from hashing plus keyword extraction with term-frequency, heading boost and first-paragraph boost (1024-dim), per-repo vector namespaces with timestamp TTL filtering, and hybrid search scoring combining normalized vector similarity (60%) with keyword/proximity/heading heuristics (40%). The hybrid-scoring and namespace-isolation patterns are directly reusable for any documentation retrieval use case without requiring an external embedding API.
A clean RepoHandler interface (getTools/fetchDocumentation/searchRepositoryDocumentation) with implementations for generic repos, three.js, and react-router that lets individual repositories or repo families expose custom tool sets, doc sources, and search strategies without forking the codebase. The generic handler plus static library-to-repo mapping covers the long tail. This is a textbook strategy seam that could be extracted for any multi-target documentation/tooling service.
Generic Cloudflare KV caching utilities with TTL jitter to prevent cache stampedes, plus specialized wrappers for URL-content caching, robots.txt caching, and fetch-doc caching. The TTL-jitter helper is a non-obvious, widely applicable pattern (shared by many production caching layers) and the module isolates KV access behind small typed functions, making it extractable into any Workers project.
A rate-limit-aware GitHub API client with retry/backoff logic, rate-limit remaining/reset tracking, and Cloudflare Analytics Engine event emission. Handles the classic GitHub 403/rate-limit failure mode correctly and is structured as an independent client that could be reused in any Workers-based GitHub integration.
Parses, caches (with TTL), and enforces robots.txt disallow rules before GitMCP fetches external URLs. Most documentation-fetching tools ignore robots.txt entirely; this module implements the policy correctly with caching to avoid repeated fetches. The pattern - policy download, parse, cache, enforce - is directly reusable for any crawler or doc-fetcher that needs to be a good citizen.