$ show bitmakerla/estela
Estela is an open-source web scraping orchestration platform (analogous to Scrapy Cloud) for deploying, scheduling, and monitoring Scrapy-based spider projects on Kubernetes. It consists of a Django REST API backend with Celery for async task management, a React/TypeScript frontend with Ant Design, and a Kafka-based queue consumer for streaming scraped data into MongoDB. Core workflows include project management, spider deployment (Docker image build via Kaniko), job execution with configurable resource tiers, cron scheduling, data persistence settings, and a metering subsystem for usage tracking. The system relies on external dependencies such as Kubernetes, Kafka, MySQL, MongoDB, Redis, and object storage (MinIO/S3). The frontend provides interactive charts, a tour overlay for onboarding, and notification management. While functional and extensible, the codebase shows signs of age (outdated Django/Celery versions, class-heavy React components, duplicate logic in cron expression handling) and has incomplete test coverage.
The LLM identified 3 files worth highlighting for reuse:
Implements the Kubernetes engine for spider job lifecycle management (create, delete, status checks). Uses a clean interface inherited from a base engine, handling pod creation, resource tier mappings, and build pipeline coordination. Could be extracted as a reusable Kubernetes job runner for other applications with minimal changes.
Handles job close metering reconciliation by comparing hourly delta slices with final job totals and generating adjustment records. Uses idempotency keys to prevent duplicate inserts. This pattern is reusable for any system needing reliable usage aggregation and correction, e.g., billing or capacity planning.
A well-structured interactive tour component that displays step-by-step guides over the UI. It supports reactive step triggers, persistent progress via localStorage, and customizable step definitions independently defined in a separate file. Could be extracted as a generic onboarding library for any React application.