Garden is a one-URL automated asset discovery and passive security scanning tool designed for authorized targets. It performs passive, read-only scans (bounded GET requests within same-origin boundaries) without active exploitation. The tool supports two primary modes: quick (anonymous) scans and authenticated coverage analysis across three fixed contexts (anonymous, user, admin). Outputs include structured Markdown reports with asset inventory, evidence, security control observations, and explicit coverage gap explanations, emphasizing that zero findings do not imply safety unless coverage is fully qualified. The architecture is a Python 3.10+ application using FastAPI for the web API, Typer for the CLI, SQLAlchemy with Alembic for database management, and Playwright for browser automation during login and inventory collection. The system follows a service layer pattern shared by CLI, Web, and API entry points, with a multi-stage pipeline (validate → collect → normalize → compare coverage → replay authorization → analyze → report) orchestrated by ScanApplicationService. External dependencies include httpx for HTTP requests, Pydantic for data validation, and Jinja2 for templating. Garden is positioned as an internal AppSec workflow for post-authentication verification of authorized application surfaces, not as a general-purpose vulnerability scanner like Burp or ZAP. It features centralized redaction of sensitive data across all outputs, an ephemeral secret store for temporary credential handling, and a plugin-based security check system with six built-in low-risk checks. The codebase includes extensive test coverage (unit, integration, e2e), a comprehensive threat model, and detailed implementation plans for versioned releases (v0.2 trusted report quality, v0.3 passive authenticated coverage).
This repository is the central specification and test suite hub for the RealWorld project (a Medium.com clone demo app). It does not contain any runnable application code. Instead, it defines the API contract via an OpenAPI spec (specs/api/openapi.yml), provides comprehensive test suites in Hurl and Bruno formats for backend validation, and a shared Playwright E2E test suite for frontend validation. It also includes a shared CSS theme (assets/theme/styles.css), documentation built with Astro/Starlight (docs/), and SVG logo generation scripts. The repository serves as the single source of truth that all RealWorld implementations (over 100 backends and frontends) must conform to. Its architecture is that of a specification and validation toolkit, not a deployable service. Key entry points are the Makefile for running tests and generating Bruno collections, and the test runner scripts in specs/api/. External dependencies include Hurl, Bruno, Playwright, Bun, and Astro.
MediaCrawler is a multi-platform social media data collection tool targeting Chinese platforms: Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu. It uses Playwright browser automation to log in (QR code, phone, cookie) and then scrapes posts, comments, and media files based on search keywords, specific post IDs, or creator IDs. The architecture is a monolith with modular platform-specific crawlers, a shared abstract base layer, and a FastAPI backend that provides a WebUI for visual control. Data can be stored in multiple formats (CSV, JSON, JSONL, SQLite, MySQL, PostgreSQL, MongoDB, Excel). Additional features include a robust media downloader with DASH support, proxy rotation, word cloud generation, and privacy-focused anonymization. The codebase demonstrates solid engineering with design patterns (factory, abstract base, mixin), good test coverage, and a clean separation of concerns, though there is some duplication across platform implementations.
Auto-Redbook-Skills is a CLI tool for creating and publishing Xiaohongshu (Little Red Book) social media notes. It takes Markdown content with YAML front matter, renders it into a cover image and multiple card images using Playwright (headless browser), and optionally publishes the note to Xiaohongshu via a built-in, reverse-engineered HTTP signature client. The tool supports 8 visual themes (e.g., sketch, neo-brutalism, botanical) and 4 pagination modes (manual separator, auto-fit, auto-split, dynamic height). The publishing pipeline is entirely self-contained within the repository, using a vendor module that implements the Xiaohongshu creator platform's authentication, signing, and upload protocols without external API services. The project is primarily a content creation automation tool, not a data research project, though it does involve some data transformation (Markdown to images).
feedgrab is a universal content aggregation tool that fetches, normalizes, and saves content from 18+ platforms (Twitter/X, Xiaohongshu, WeChat, YouTube, Bilibili, GitHub, HackerNews, Medium, Reddit, Weibo, Douyin, Telegram, RSS, Feishu, Kdocs, Zsxq, FlowUs, and more) into Obsidian-compatible Markdown files with YAML frontmatter. It operates as a CLI tool, a Python library, and an MCP server, featuring a modular architecture with per-platform fetchers that implement a multi-tier fallback strategy (direct API → CDP browser reuse → Playwright launch → Jina Reader) to maximize resilience against anti-scraping measures. The system includes robust utilities for deduplication, media download, Groq Whisper transcription, paywall bypass, and cookie rotation for authenticated platforms. Originally developed as a 'vibe coding' personal project, it has evolved into a well-structured application with a service layer, typed configuration, and comprehensive CLI commands (login, doctor, batch search, setup wizard). It is actively maintained, with frequent commits addressing platform API changes and adding new features.
HeadlessX is a self-hosted web scraping and browser automation platform with anti-detection capabilities. It provides a Next.js 16 dashboard, an Express 5 API with Prisma/PostgreSQL, a BullMQ queue worker, and a published CLI. The core value proposition is anti-detection scraping via the headfox-js package (Camoufox-compatible Firefox bundles), fingerprint generation, CAPTCHA solving, and LLM-friendly markdown output. The system supports multiple operators: website scraping/crawling/mapping, Google AI Search, Tavily search, Exa search, and YouTube extraction. It includes sidecar services for HTML-to-Markdown (Go) and YouTube extraction (Python FastAPI). The architecture is a monorepo with Nx orchestration, Docker Compose deployment, and an MCP endpoint for AI assistant integration. External dependencies include Redis, Postgres, Playwright, MaxMind GeoIP, and external APIs (Tavily, Exa, Google). The codebase is well-structured with clear separation of concerns, modern TypeScript, and reusable patterns in both frontend and backend.
CloakBrowser is a commercial stealth browser automation library that wraps Playwright (Python, JavaScript, .NET) with a custom patched Chromium binary. Its core purpose is to evade bot detection by spoofing browser fingerprints, providing human-like interaction (Bezier mouse moves, realistic typing with typos, smooth scrolling), and managing licensed binary downloads with cryptographic verification. The architecture is a multi-language SDK sharing identical logic across Python, TypeScript, and C#, with a common human interaction engine, proxy/GeoIP integration, and license enforcement via denial files and server validation. Key entry points are the `launch()` and `humanize()` APIs, CLI diagnostics (`info`, `login`, `install`), and the `cloakserve` CDP multiplexer. External dependencies include Playwright/Puppeteer, MaxMind GeoLite2 database, Ed25519 signature verification (BouncyCastle), and a license validation server. The codebase demonstrates high engineering discipline with extensive cross-language parity tests and security-conscious binary distribution, but carries drift risks from triple-language duplication and reliance on Playwright internal APIs.
This repository implements a production-grade Google Maps business data extraction platform with multiple execution modes: CLI, web UI, REST API, AWS Lambda, and a self-hosted SaaS edition. The core scraping engine uses Playwright for browser automation and also supports a stealth HTTP fast mode. It extracts 36+ structured fields per business (name, address, phone, website, reviews, ratings, emails, coordinates, hours, etc.) and provides deduplication, grid-based area coverage, email extraction via website visits, and review parsing. The architecture is modular with packages for scraping jobs, runners (file, database, web, Lambda, SaaS worker), output writers (CSV, JSON, PostgreSQL, S3, custom plugins), job queuing via River, and a comprehensive admin dashboard with cloud provisioning on DigitalOcean and Hetzner. The system is designed for both single-user CLI use and multi-tenant SaaS deployments with authentication, 2FA, API key management, and scalable worker pools. The codebase demonstrates strong Go engineering practices with well-defined interfaces, dependency injection, and test coverage. Notable reusable components include cryptography utilities, HTTP server middleware, rate limiting, and a CLI prompt library. However, the repository has some maintenance concerns: two separate database migration sets for legacy and SaaS modes, in-memory deduplication that doesn't persist across restarts, committed test SSH keys in the `local/` directory, and a large dependency surface that could introduce version conflicts. The scraper logic is inherently brittle due to reliance on Google Maps internal data structures, which are subject to change without notice. For AI agent consumption: This is a powerful off-the-shelf scraping tool that can be deployed as a Docker container, AWS Lambda, or managed SaaS. It provides a complete API and web interface for managing scraping jobs, monitoring progress, and exporting data. The SaaS edition includes worker auto-provisioning, making it suitable for large-scale lead generation operations. The code is actively maintained with recent commits addressing performance and feature enhancements.
This repository is a TypeScript monorepo for an AI-powered autonomous web browsing agent called 'open-browser'. It consists of three packages: core library (open-browser), CLI (@open-browser/cli), and sandbox (@open-browser/sandbox). The core library orchestrates an agent loop that uses Playwright to control a Chromium browser, extracts DOM state via CDP, sends it to an LLM (OpenAI, Anthropic, Google) which decides actions in structured JSON, and executes those actions via a CommandExecutor. The system supports vision screenshots, extended thinking, compact mode, planning, evaluation, replay recording, and MCP server integration. The CLI provides commands for running agents, interactive sessions, and direct browser manipulation. The sandbox provides resource-limited execution. Key entry points include the Agent class, Viewport browser session, CommandExecutor, and BridgeServer. External dependencies include Playwright, Vercel AI SDK, Zod, Turndown, and various LLM provider SDKs. The architecture follows a modular monorepo with clear separation of concerns, event-driven communication via EventHub, and a guard/watchdog pattern for browser lifecycle management. The codebase demonstrates strong engineering practices with extensive use of Zod schemas for structured outputs, a command catalog pattern, and comprehensive telemetry and cost tracking.
This repository provides a Node.js HTTP service that generates TikTok API signatures (X-Bogus and X-Gnarly) required to access TikTok's data endpoints. It uses Puppeteer to run a headless Chromium browser, injects a bundled copy of TikTok's web SDK, and computes signatures by intercepting fetch requests. The service also includes a pure Node.js implementation for X-Gnarly using ChaCha20. It exposes POST /signature and POST /fetch endpoints, manages a request queue, and automatically refreshes the browser session after 500 signatures or 30 minutes. The repository includes example scripts for fetching user videos, comments, trending content, and search results, demonstrating typical TikTok data scraping workflows. While the core functionality is well-implemented, the reliance on static SDK copies and a heavy browser dependency introduces maintenance risks as TikTok updates its anti-scraping measures.
CF-Clearance-Scraper is a Python CLI tool designed to solve Cloudflare challenges (JavaScript, managed, and interactive) and extract the cf_clearance cookie for bypassing anti-bot protections. The application uses Zendriver for headless browser automation, dynamically generates realistic Chrome user agents, handles proxy authentication via CDP events, and provides output formats including raw cookies, cURL/Wget/aria2 commands, or JSON storage. The architecture is monolithic with a single entry point (main.py) featuring clean separation of concerns: argument parsing, browser orchestration, challenge detection/solving, and output formatting. Key strengths include robust user agent fallback mechanics and proxy handling, though the Cloudflare-solving logic is tightly coupled to Zendriver and challenge-specific heuristics.
Botright is a Python library built on Playwright that provides advanced browser automation with stealth features to evade bot detection. It includes proxy management, fingerprint spoofing using real browser profiles (chrome-fingerprints), human-like mouse movements, and CAPTCHA solving for hCaptcha, reCAPTCHA, and Geetest (though Geetest support is currently disabled). The library wraps Playwright's API with a custom layer that intercepts browser events, modifies user agent, platform, and other fingerprinting vectors, and injects anti-detection scripts. It supports both headless and headed modes, proxy rotation, and response caching. The codebase is well-structured but has commented-out modules indicating incomplete transitions (e.g., Geetest).
ClawCloud-Run is a GitHub Actions-based automation script designed to automatically log into ClawCloudRun cloud service to maintain account activity. The system handles GitHub OAuth authentication, device verification, and two-factor authentication (via GitHub Mobile or Telegram-delivered TOTP codes). After login, it navigates to the console and apps pages to keep the session active, extracts the updated session cookie, and either updates it in GitHub Secrets or sends it via Telegram for manual update. The architecture consists of a workflow triggering a Python script that uses Playwright for browser automation, requests for API interactions, and pynacl for secret encryption. Key entry points are the GitHub Actions workflow (scheduled every 5 days) and the Python script's main execution loop.