Estela is an open-source web scraping orchestration platform (analogous to Scrapy Cloud) for deploying, scheduling, and monitoring Scrapy-based spider projects on Kubernetes. It consists of a Django REST API backend with Celery for async task management, a React/TypeScript frontend with Ant Design, and a Kafka-based queue consumer for streaming scraped data into MongoDB. Core workflows include project management, spider deployment (Docker image build via Kaniko), job execution with configurable resource tiers, cron scheduling, data persistence settings, and a metering subsystem for usage tracking. The system relies on external dependencies such as Kubernetes, Kafka, MySQL, MongoDB, Redis, and object storage (MinIO/S3). The frontend provides interactive charts, a tour overlay for onboarding, and notification management. While functional and extensible, the codebase shows signs of age (outdated Django/Celery versions, class-heavy React components, duplicate logic in cron expression handling) and has incomplete test coverage.
LanzouAPI is a single-file PHP script that acts as a parsing proxy for Lanzou (θε₯δΊ) cloud storage share links. It accepts a share URL and optional password, scrapes the Lanzou page, handles anti-scraping challenges (including ESA and acw_sc__v2 cookie generation), and returns a JSON response containing the file name, size, and a direct download URL. Optionally, it can redirect directly to the download. The script includes a local file-based cache with configurable TTL to reduce requests and avoid rate limiting. It also provides a custom suffix replacement feature for download URLs. The codebase is a monolith with no framework, no tests, and no dependency management; all logic resides in a single index.php file. While functional and actively used (as evidenced by the health-check workflow), the architecture is tightly coupled and would require significant refactoring for reuse beyond this specific Lanzou API use case.
Reels-AutoPilot is a Python automation application that scrapes short-form video (Instagram Reels and YouTube Shorts) from configured creator accounts and reposts that content to the user's own Instagram account, including optional story cross-posting. The architecture is a small monolith of loosely coupled scripts sharing a SQLite database: app.py is the main orchestrator running an infinite scheduling loop, reels.py and shorts.py scrape content via instagrapi/yt-dlp, poster.py uploads clips and stories, remover.py purges posted files, and db.py/helpers.py provide SQLAlchemy persistence plus a config system where SQLite values override static defaults in config.py. A Rich-based terminal dashboard (dashboard.py) shows live posting status, and start.py is an interactive CLI setup wizard. The core data flow is: scrape β download to downloads/ β insert metadata row into the 'reels' table as unposted β poster loop picks the oldest unposted row β upload as an Instagram clip β mark posted β optionally trim to 15s and post as a story with mention/link/hashtag/media stickers β remover deletes the local file after a configurable interval. External dependencies are the unofficial Instagram client instagrapi, the YouTube Data API v3 with yt-dlp, moviepy for video trimming, SQLAlchemy, and Rich. The code is functional but rough: reels.py has a filter bug (a tuple instead of a boolean condition) that lets non-reel media into the database; dashboard.py compares string config values to integers so status toggles always render as 'Off'; start.py validates the wrong variable in two input loops; exceptions are silently swallowed; and Instagram credentials are stored in plaintext in both config.py and the SQLite DB, with database/sqlite.db committed to the repository (a credential-leak risk). There are no tests or CI, and the README advertises features (scheduling, multi-account, auto-hashtags, analytics) that do not exist in the code. Dependency versions are somewhat dated, with moviepy 1.0.3 (circa 2020) the most notable lag. The YouTube channel-resolution helpers and the instagrapi story-sticker composition are the most genuinely reusable pieces.
This repository contains a monolithic Python CLI tool for downloading Udemy courses, including DRM-protected video content. The core logic in main.py orchestrates authentication (bearer token or browser cookies), fetches course curriculum via Udemy's private API, and downloads lectures, subtitles, assets, quizzes (normal and coding assignments), practice assignments, and role-play exercises. It relies heavily on external tools: yt-dlp for DASH/HLS segment downloading, aria2c for accelerated downloads, ffmpeg for muxing and video transcoding, and shaka-packager for DRM decryption. The code is functional but lacks modularity, with global state, sparse error handling, and a monolithic main script. The standout component is mp4parse.py, a well-structured MP4 box parser that extracts PSSH headers for Widevine key identification, which could be extracted and reused independently. The project is no longer actively maintained as stated in its README, and some dependencies may be outdated.