$ show watercrawl/WaterCrawl
WaterCrawl is a full-stack web crawling and data extraction platform, built as a SaaS-style multi-tenant application. The backend is implemented in Django 5.1 with Django REST Framework, providing REST APIs for crawl requests (single/batch URL), web search (Google Custom Search integration), sitemap generation, and proxy server management. Asynchronous task execution is handled by Celery workers that invoke Scrapy spiders (SiteScrapper, GoogleCustomSearchScrapper, SitemapScrapper) and optionally use Playwright for JavaScript rendering. Real-time status updates are pushed to the frontend via Redis Pub/Sub and Server-Sent Events (SSE). The frontend is a React SPA (with TypeScript, Tailwind CSS, react-hook-form) that provides dashboard pages for configuring and monitoring crawl/search/sitemap jobs, managing teams and API keys, viewing usage statistics, and handling subscription billing through Stripe. Authentication supports JWT (simplejwt) and OAuth (Google, GitHub), with email verification and team invitation flows. The application stores extracted content (markdown, HTML, attachments) in MinIO or S3-compatible storage, and metadata in PostgreSQL.
The LLM identified 4 files worth highlighting for reuse:
Provides Fernet-based symmetric encryption/decryption utilities (make_key, encrypt, decrypt, encrypt_str, decrypt_str). Self-contained module that can be reused in any Python project requiring simple field-level encryption.
Implements a RedisLock context manager using a Lua script for atomic acquire/release. Provides a clean, reusable pattern for distributed locking in Redis-backed applications.
Contains HtmlFilter (extract main content via tags, CSS selectors, custom filters), HtmlToMarkdown (conversion using markdownify), and ImageToBase64 (inline image handling). Self-contained module for transforming raw HTML into structured Markdown content.
A dynamic form renderer that builds UI from JSON Schema definitions. Supports a pluggable widget registry (Text, Number, Select, Switch, Radio, TextArea, Checkbox, JsonEditor) and nested fields (ArrayField, ObjectField). Demonstrates a clean separation of schema parsing, field rendering, and validation – easily extractable for any React project needing schema-driven forms.