TubeArchivist is a self-hosted YouTube media server that downloads, indexes, and serves YouTube video content. In production, it runs as a Docker-based stack with a Django REST API backend, React/TypeScript frontend, Celery task queue, Redis broker, and Elasticsearch for metadata storage and full-text search. The system subscribes to YouTube channels and playlists via yt-dlp, extracts comprehensive metadata (including subtitles, comments, SponsorBlock segments, and thumbnails), and provides a web UI for browsing, searching, streaming video, and managing downloads. It supports multiple authentication modes (single user, LDAP, forward proxy), scheduled tasks for rescanning subscriptions and backups, and integrates with Apprise for notifications and a membership/SponsorBlock API for ad-skipping. The architecture follows a modular Django app pattern (video, channel, playlist, download, task, appsettings, user, stats) with consistent use of base views, serializers, and search processors. The frontend uses Zustand for state management and a custom API client with CSRF handling. The codebase is actively maintained with comprehensive test coverage and CI/CD pipelines.
IntelOwl is a production-grade, open-source threat intelligence platform that automates the analysis of observables (IPs, domains, URLs, hashes) and files (PE, PDF, APK, etc.) through a pluggable plugin system. The backend is built with Django REST Framework, Celery for asynchronous task execution, and Django Channels for real-time WebSocket updates. The frontend is a React single-page application using react-router, react-flow for pipeline visualizations, and Zustand for state management. The system supports multi-tenancy, organizations, role-based access, and integrates with dozens of external threat intelligence APIs and open-source analysis tools via containerized microservices. The architecture is centered around a plugin framework with five plugin types: analyzers (file and observable), connectors (push results to external systems), visualizers (render reports), pivots (create child jobs from analysis results), and ingestors (pull threat feeds). Playbooks bundle these plugins into repeatable analysis workflows. A data model manager normalizes analyzer outputs into structured evaluations with reliability scores, and a user event system allows manual verdict overrides with decay logic. The platform also includes an AI chatbot assistant powered by a local Ollama LLM and LangGraph, enabling natural-language queries and one-click analysis launches. Key operational features include a comprehensive REST API, token-based authentication with optional OAuth/LDAP/RADIUS, Elasticsearch for BI reporting, and Docker-based deployment with separate containers for specialized analyzers. The codebase is actively maintained with extensive unit and integration tests, CI/CD pipelines, and a large community (4695 stars). While the migration system is somewhat repetitive, the overall architecture is clean, modular, and demonstrates strong software engineering practices.
django-celery-beat is a Django library that provides a database-backed scheduler for Celery Beat, allowing periodic task schedules to be stored and managed in a Django database. It offers models for interval, crontab, solar, and clocked schedules, with a full Django admin interface for CRUD operations. The core scheduler (`DatabaseScheduler`) reads enabled periodic tasks from the database and manages their execution, with change detection via a dedicated `PeriodicTasks` model. The codebase is production-stable, well-tested, and actively maintained, supporting recent Django (6.1) and Python (3.14) versions.
This is a comprehensive Cookiecutter template for generating production-ready Python projects. It is not a running application itself but a project generator that produces applications, libraries, or services based on user-selected features. The template includes sophisticated post-generation hooks that surgically remove unused files and configure dependencies, Docker Compose services, CI/CD workflows, and documentation based on the chosen options. It enforces modern Python practices: uv for package management, Ruff for linting/formatting, mypy for type checking, pytest for testing, and Pydantic Settings for configuration. The generated projects can include FastAPI, Celery, SQLAlchemy, QuasiQueue, aiocache, Jinja2, Typer CLI, Docker, and GitHub Actions. The template itself demonstrates high craftsmanship with clean Jinja2 templating, conditional logic, and a well-organized file structure. It is actively maintained and uses current tooling (uv, Ruff, etc.), indicating low technical drift.
PrivateGPT is a production-grade, private AI assistant platform that provides RAG (Retrieval-Augmented Generation) over ingested documents, tool/function calling, sandboxed code execution, web search, MCP (Model Context Protocol) integration, skill management, and multimodal support. It is built as a FastAPI server with background workers (Celery and ARQ) for asynchronous ingestion and task processing, and exposes a Claude API-compatible streaming interface. The architecture is modular: a core chat engine with an interceptor pipeline for request/response processing, a pluggable component system with registries for LLMs, embeddings, readers, vector stores, and sandboxes, and a two-tier scheduling system (local and distributed) for tool and chat execution. The codebase demonstrates strong engineering practices with dependency injection, registry patterns, and composable interceptors, but its dual scheduling backends, extensive monkey-patching of third-party libraries, and heavy reliance on llama-index introduce maintenance complexity and technical drift. Reusable highlights include the context stack builder for dynamic system prompt assembly, the interceptor chain pattern, the sandbox abstraction for secure code execution, and the citation parser with incremental streaming support. The project is actively maintained by Zylon and powers their enterprise offering, with 57k+ stars and frequent releases.
CourtListener is a comprehensive legal research platform operated by Free Law Project. It aggregates U.S. court opinions, dockets, oral arguments, and judge profiles from multiple sources including PACER/RECAP, Harvard Caselaw, Columbia, state courts (Florida, Texas, New York), and SCOTUS. The system provides full-text search via Elasticsearch, real-time and scheduled alerts, REST API with OAuth2, webhook notifications, citation analysis, and audio transcription. It also includes tools for bulk data import, data cleaning, and embedding generation for LLM training. Architecturally, it is a Django 6.0+ monolith with a Celery task queue (Redis broker), PostgreSQL database, Elasticsearch search engine, and S3-compatible storage. The frontend uses Django templates with Alpine.js and some React components, built via Webpack. The system runs as multiple Docker containers (web, Celery workers, scrapers, daemons). Key data flows include nightly PACER scraping, email-based RECAP document ingestion, citation extraction using eyecite, and state-specific docket merging pipelines. The system is heavily event-driven with pgtrigger-based audit trails and Redis-based throttling. While primarily an application, it has a strong data research character: extensive ETL pipelines, machine learning integration (LLMs for text extraction, OpenAI Whisper for transcription), citation network analysis, and state court data modeling. The codebase demonstrates high craftsmanship in reusable patterns like the CeleryThrottle, tiered caching, and generic merger classes, but also carries some technical debt from many SQL migrations and tightly coupled scrapers.
Onyx is a full-stack AI enterprise search and chat platform. The backend is a Python/FastAPI service with Celery workers, PostgreSQL, Redis, and a vector search engine (Vespa/OpenSearch). It ingests documents from 50+ external connectors (Google Drive, Slack, Confluence, Jira, SharePoint, etc.), runs a chunking/embedding/indexing pipeline, and serves RAG chat and search to multiple frontends: Next.js web app, React Native/Expo mobile app, Tauri desktop wrapper, Go CLI, an embeddable web widget, and a Terraform provider. It also includes an MCP server, deep research flows, sandboxed code-execution sessions (Craft), prompt caching, SSO/OAuth/SCIM multi-tenancy, rate limiting, and an admin analytics console. The repository has a hybrid character because it contains substantial data-engineering machinery (indexing pipelines, chunkers, embedding providers, evals) but the dominant identity is a production application rather than a research project.
Read the Docs is a Django-based web application for building, hosting, and serving documentation from version control repositories (GitHub, GitLab, Bitbucket). It supports Sphinx, MkDocs, and other documentation generators, with automated builds triggered by webhooks, versioning (semver, calver, custom), custom domains, subprojects, translations, and a full-text search powered by Elasticsearch. The platform includes a multi-tenant organization system with teams, invitations, and Stripe-based subscriptions for commercial features like concurrent builds and private documentation. Architecturally, it is a monolithic Django application with multiple apps (projects, builds, search, analytics, oauth, embed, proxito, etc.) communicating via Celery task queues for async operations (builds, indexing, email). It exposes REST APIs (v2 and v3) for programmatic access, and serves documentation through a custom proxito middleware that handles domain resolution, caching, redirects, and CDN integration. The codebase demonstrates strong engineering practices with extensive use of mixins, class overrides via SettingsOverrideObject, structured logging with structlog, and a comprehensive test suite. Key features include OAuth-based repository synchronization, automation rules for version management, an embed API for content fragments, file tree diff for pull request previews, audit logging, and spam detection. The system is designed for extensibility, with many components replaceable through Django settings, and relies on external services like AWS S3 for storage, Elasticsearch for search, and Stripe for payments.
WaterCrawl is a full-stack web crawling and data extraction platform, built as a SaaS-style multi-tenant application. The backend is implemented in Django 5.1 with Django REST Framework, providing REST APIs for crawl requests (single/batch URL), web search (Google Custom Search integration), sitemap generation, and proxy server management. Asynchronous task execution is handled by Celery workers that invoke Scrapy spiders (SiteScrapper, GoogleCustomSearchScrapper, SitemapScrapper) and optionally use Playwright for JavaScript rendering. Real-time status updates are pushed to the frontend via Redis Pub/Sub and Server-Sent Events (SSE). The frontend is a React SPA (with TypeScript, Tailwind CSS, react-hook-form) that provides dashboard pages for configuring and monitoring crawl/search/sitemap jobs, managing teams and API keys, viewing usage statistics, and handling subscription billing through Stripe. Authentication supports JWT (simplejwt) and OAuth (Google, GitHub), with email verification and team invitation flows. The application stores extracted content (markdown, HTML, attachments) in MinIO or S3-compatible storage, and metadata in PostgreSQL.
Yeti is a threat intelligence platform that ingests, enriches, correlates, and stores cyber threat observables, entities, indicators (YARA, Sigma, Suricata, regex, forensic artifacts), and DFIQ investigative questions. It provides a FastAPI-based REST API for CRUD operations, graph traversal, semantic search (via ChromaDB), and RBAC. The system uses ArangoDB as its primary graph/document database, Redis for message brokering, and Celery for background task scheduling. It includes a plugin architecture for feeds (30+ public sources like MITRE ATT&CK, CISA KEV, MISP, AbuseIPDB) and analytics (Shodan, Censys, VirusTotal, etc.). The codebase is well-structured with clean abstractions (ArangoYetiConnector, event system, polymorphic schemas, migration framework) but contains deprecated plugins and some configuration drift. It is primarily an application, not a data research project.
DefectDojo is an open-source application security posture management (ASPM) and vulnerability management platform. It ingests scan results from over 100 security tools (SAST, DAST, SCA, container, cloud, secrets, infrastructure) via a consistent parser interface, normalizes findings, deduplicates them, and tracks remediation workflows across engagements and products. The system is a Django monolith with a REST API (Django REST Framework), Celery async workers for imports, SLA enforcement, and JIRA/GitHub sync, PostgreSQL for persistence, and Valkey/Redis as the message broker. Key entry points include the Django admin UI, DRF API viewsets, and Celery task dispatchers. External dependencies are PostgreSQL, Redis/Valkey, and optional integrations with JIRA, Slack, Microsoft Teams, and SSO providers (LDAP, OAuth2, SAML). The repository includes a Pro edition layer with additional features (custom RBAC, audit logging, priority SLA) that coexist with the open-source core via feature flags and conditional imports. The codebase demonstrates strong engineering with a well-defined parser factory pattern, prefetch mixins for API efficiency, SSRF-safe HTTP clients, and a request-level caching system, though it carries some technical debt from an ongoing V3 location model migration and numerous deprecated endpoints slated for removal.
SynoCommunity/spkrepo is a production-grade web application for hosting and distributing Synology package (.spk) repositories. It provides a full DSM-compatible catalog API for NAS devices, an admin interface for package lifecycle management (upload, sign, activate, archive), user registration with role-based access, and background task processing via Celery. The system is built with a hexagonal (ports-and-adapters) architecture, separating domain logic from Flask web transports, SQLAlchemy persistence, and Celery task execution. It manages package metadata, versioning, firmware compatibility, and download statistics. The codebase is well-structured, with extensive test coverage including import isolation enforcement, Alembic migrations for PostgreSQL and SQLite, and CI/CD via GitHub Actions. Dependencies include Flask, Celery, PostgreSQL, Redis, Bootstrap, and Chart.js. The architecture is clean and modular, making it suitable for both small community repos and larger installations.
This repository is an AI-driven story-to-video production platform. It is a full-stack Django 3.2 backend with Vue 2 frontend, Celery task queue, Redis pub/sub, and Docker deployment. The system ingests user story ideas and orchestrates a multi-stage pipeline: rewrite → storyboard → image generation (text2image/image2video) → camera movement → video generation → Jianying (剪映) draft export. All AI capabilities are consumed via external APIs (OpenAI, ComfyUI, Volcano Engine, etc.) through a pluggable AI client abstraction layer. Key entry points include RESTful APIs (DRF), SSE streaming for real-time progress, MCP (Model Context Protocol) JSON-RPC endpoints for agent integration, and a Vue-based canvas UI for visual workflow editing. The architecture follows a modular Django app pattern with clear separation: core infrastructure (pipeline orchestration, AI clients, Redis streaming), domain apps (projects, prompts, models, workflows, content, agent, MCP), and a frontend SPA. External dependencies include Redis, Celery, MySQL/PostgreSQL, external AI vendor APIs, and optional Jianying draft generation. The codebase demonstrates strong engineering practices with reusable patterns for AI client abstraction, Redis pub/sub streaming, MCP tool registration, and pipeline stage processors, but has notable technical debt: Django 3.2 (EOL), mixed async/gevent, Vue 2 (EOL), and some closed-source compiled components.
Jellyfish is an open-source, full-stack AI short drama studio application. It provides an end-to-end production pipeline for generating vertical short dramas from scripts: script parsing (division, element extraction, consistency checking), storyboard generation (shot breakdown, asset linking, camera language), image/video generation via multiple AI providers (OpenAI, Volcengine), and async task orchestration with Celery. The backend is built with FastAPI, SQLAlchemy 2.0 async, and Redis; the frontend uses React 18, TypeScript, Ant Design, and Vite. The application is production-oriented with Docker Compose deployment, GitHub Actions CI, and automated OpenAPI client generation. It emphasizes visual consistency through global seed, style inheritance, and asset reuse. While primarily an application, it includes a significant amount of AI/LLM agent code for script analysis, giving it a hybrid character with research-oriented structured extraction pipelines.
iKy (I Know You) is a modular OSINT (Open Source Intelligence) aggregation platform built with FastAPI, Celery, Redis, and an Angular frontend. The system ingests a username or email and concurrently executes 26+ parallel tasks across social media, data breaches, search engines, and dark web sources, returning a structured JSON contract (module, param, validation, raw, graphic, profile, timeline, tasks) consumed by a visual graph/timeline UI. Key architectural patterns include a dynamic module registry that maps route names to Celery task paths, a decorator-based task wrapper (@iky_task) that centralizes dev-mode golden-file bypass, error classification, and performance logging, and a pluggable search provider orchestration layer. The codebase demonstrates a strong separation of concerns between API routing, task execution, and data enrichment, but suffers from fragile scraping logic (LinkedIn RSC parsing, TikTok SSR regex), heavy reliance on browser session cookies, and several deprecated external APIs (Twitter v1.1, FullContact). The project is actively maintained with an extensive test suite (~4K lines) and Docker Compose deployment, though many tests are heavily mocked and may not detect real API drift.
reNgine is a Django-based automated security reconnaissance and attack-surface management platform. It orchestrates 20+ external security tools (Subfinder, Amass, Naabu, Nuclei, FFUF, GoSpider, Dalfox, WAFW00f, EyeWitness, theHarvester, h8mail) via Celery task chains to perform subdomain discovery, port scanning, HTTP crawling, directory fuzzing, OSINT gathering, technology fingerprinting, WAF detection, and vulnerability scanning. Results are parsed from tool outputs (nmap XML, nuclei JSON, dalfox JSON, ffuf JSON), persisted in PostgreSQL, and exposed through a Django REST Framework API plus a jQuery/Bootstrap frontend with server-side DataTables, dashboards, and WeasyPrint PDF reports. The system supports multi-project tenancy, role-based access control (sys_admin, penetration_tester, auditor), scheduled scans via Celery Beat, and deep HackerOne bug-bounty integration for program import, bookmarking, and vulnerability report submission. It also integrates LLMs (OpenAI and local Ollama) for vulnerability description enrichment and attack-surface suggestions. Architecturally it is a monolithic Django application with four feature apps (dashboard, scanEngine, startScan, targetApp, recon_note) and a Docker deployment composed of web, celery, celery-beat, redis, postgres, nginx, and ollama services. The orchestration core, web/reNgine/tasks.py (~3000 lines), composes Celery groups/chords/chains, executes shell commands against external tools, parses their structured output, and persists scan results. The frontend is server-rendered Django templates with heavy inline JavaScript and a supplementary DRF API layer in web/api/views.py. Although it collects and aggregates large volumes of scan-derived data, this is fundamentally an operational security application, not a data-research project. The codebase is actively patched (recent CVE fixes for command injection and XSS) and feature-rich, but carries notable technical debt: an end-of-life Django 3.2 base, old vendored JS plugins (DataTables 1.10.16, Lightbox from 2018), hardcoded container filesystem paths, a monolithic task module mixing orchestration/parsing/persistence, and dead or commented code. The strongest reusable pieces are the RengineTask Celery base class (standardized task logging, caching, and notifications), the dual-provider LLM abstraction with OpenAI/Ollama backends, and a custom server-side datatable filtering language applying operators (=, >, <, !, &, |) across subdomain, endpoint, and vulnerability views. It is best characterized as effective, pragmatic, security-tool engineering rather than clean, academically elegant architecture.
DeepV-Ki is an AI-powered wiki generator for code repositories that combines a Next.js frontend with a FastAPI backend. The system allows users to input a repository URL and automatically generates documentation, architecture diagrams (Mermaid), and provides a RAG-based Q&A interface. It features multi-LLM provider support (OpenAI, Google Gemini, Azure, AWS Bedrock, Ollama), GitLab/GitHub/Bitbucket integration, asynchronous task processing with Celery, and multi-layer caching. The architecture follows a monorepo structure with clear separation between frontend (React/Next.js) and backend (Python/FastAPI) components, communicating via REST APIs. The system includes sophisticated authentication (SSO, session management), progress tracking, and real-time WebSocket communication for streaming AI responses.
Estela is an open-source web scraping orchestration platform (analogous to Scrapy Cloud) for deploying, scheduling, and monitoring Scrapy-based spider projects on Kubernetes. It consists of a Django REST API backend with Celery for async task management, a React/TypeScript frontend with Ant Design, and a Kafka-based queue consumer for streaming scraped data into MongoDB. Core workflows include project management, spider deployment (Docker image build via Kaniko), job execution with configurable resource tiers, cron scheduling, data persistence settings, and a metering subsystem for usage tracking. The system relies on external dependencies such as Kubernetes, Kafka, MySQL, MongoDB, Redis, and object storage (MinIO/S3). The frontend provides interactive charts, a tour overlay for onboarding, and notification management. While functional and extensible, the codebase shows signs of age (outdated Django/Celery versions, class-heavy React components, duplicate logic in cron expression handling) and has incomplete test coverage.
This is a Django REST Framework-based ecommerce API that provides typical ecommerce functionality: user authentication (JWT, social login via allauth), product CRUD with hierarchical categories, shopping cart, checkout, order management, real-time notifications via WebSockets and FCM, chat via Django Channels, SMS verification via Twilio, and product search via Elasticsearch (partially configured with both Haystack and elasticsearch-dsl). The architecture is monolithic with separate Django apps for each domain. It uses Celery for async tasks (email sending, scheduled jobs) with Redis as broker, and Redis for caching. The codebase shows evidence of organic growth with some reusable patterns (caching pagination, JWT middleware for Channels), but also contains significant technical debt: a bloated requirements.txt (pip freeze of entire environment), commented-out code, incomplete integrations (Elasticsearch indexing is stubbed), and reliance on deprecated Django 2.2 APIs (ugettext_lazy). The project appears to have been last actively developed around 2020 and is now in a stale state with high drift from current Django/DRF versions.