$ show freeCodeCamp/devdocs
DevDocs is a web application that aggregates and serves API documentation for hundreds of programming languages, frameworks, and tools in a unified, searchable, offline-capable interface. The backend is a Ruby/Sinatra server that provides the static assets and a minimal API, while the frontend is a custom JavaScript single-page application with client-side search, keyboard navigation, and a service worker for offline access. The core data generation pipeline consists of Ruby scrapers that fetch documentation pages from official sources, apply a chain of filters (HTML cleaning, entry extraction, URL normalization, image optimization), and produce normalized JSON indexes and cached pages stored on disk. The architecture is largely monolithic with a strong separation between the scraping pipeline (lib/docs) and the frontend (assets/javascripts). While primarily an application, it contains a substantial data engineering component in its scraper/filter system, which is reusable and generalizable for other documentation aggregation projects.
The LLM identified 4 files worth highlighting for reuse:
Implements a composable filter pipeline that processes HTML documents through a stack of filter classes. Each filter can modify the document, add entries, or extract metadata. The stack supports insertion, removal, and ordering of filters, and runs them in sequence. This pattern is cleanly isolated and could be extracted to build any document transformation pipeline.
Base class for scraping documentation from URLs. Handles HTTP requests with rate limiting, retries, and response caching via a HAR-based cache. Subclasses define version-specific URLs and customize the scraping behavior. The rate-limiting and caching logic are reusable for any web scraping project that needs polite fetching.
A robust IndexedDB abstraction that manages a single database connection with queued operations, schema versioning, graceful fallback to network when IndexedDB is unavailable, and localStorage migration for legacy data. It handles corruption detection, transaction abortion recovery, and multiple stores for pages and indexes. This module is independently reusable as a resilient client-side storage layer.
A filter that converts documentation images (PNG, GIF, JPEG) to WebP format, optionally embeds small images as base64 data URIs, and replaces image URLs. It uses image_optim for lossless compression and caches results in the document. This component solves a common problem in documentation aggregation: reducing bandwidth and storage overhead while maintaining visual fidelity.