photoscrawl is a macOS CLI tool that crawls, archives, and analyzes Apple Photos libraries. It extracts metadata, faces, search indexes, and performs classification using local multimodal models (Ollama/OpenAI). The tool stores observations and evidence in a local SQLite database (photos.sqlite) rather than final truth, enabling later hypothesis generation for people, trips, and places. It provides subcommands for crawling, importing, classifying, searching, ranking, similarity, place backfill, and evaluation card generation. Architecture: The system is a Go CLI with a modular internal structure. It uses a crawlkit shared library for crawler mechanics, control, output, and store. The Apple Photos library is accessed via PhotoKit (cgo/Objective-C bridges) and direct SQLite snapshots. Data flows: snapshot Apple Photos database -> import metadata, faces, search indexes -> classify assets (metadata observations, then optional local model inference) -> store evidence in archive. Place backfill uses a subprocess bridge for reverse geocoding with rate limiting. Eval cards prepare images and run Ollama models for evaluation. Key entry points: cmd/photoscrawl/main.go (CLI), internal/archive/crawl.go (crawl logic), internal/archive/classify.go (classification queue), internal/place/backfill.go (batch geocoding), internal/evalcard/evalcard.go (model evaluation). External dependencies: Apple Photos (macOS), SQLite, Ollama/OpenAI (optional), crawlkit, Go standard library. The codebase demonstrates strong engineering practices with comprehensive tests, CI/CD, vulnerability scanning, and schema migrations. It is actively maintained with frequent dependency updates and a focus on privacy (all model calls local by default, no private data in repo).
EasySpider is a visual, no-code web scraping tool that combines an Electron desktop application, a Chrome browser extension, and a Python execution engine. Users design scraping workflows via a flowchart-based UI (Vue.js) where they define steps like opening pages, clicking elements, extracting data, looping, and conditional branching. The task definitions are saved as JSON graphs. The Python backend (ExecuteStage) loads these graphs and drives Chrome using Selenium with an undetected-chromedriver fork to evade bot detection. Extracted data can be written to CSV, Excel, JSON, or MySQL. The tool also includes OCR support (ddddocr), custom JavaScript/Python execution, and Cloudflare bypass capabilities. The architecture is a hybrid desktop app with a local HTTP server (Express) that spawns executor processes and communicates via WebSocket. The Chrome extension assists in element selection and XPath generation. While the tool is functional and popular (44k+ stars), the codebase exhibits significant duplication (e.g., identical task files across platform directories, duplicated Bootstrap and content scripts), hardcoded paths, monolithic executor scripts, and mixed-language strings. Several experimental modules (LLM, federated learning) are present but not integrated. The project is actively maintained with recent commits and packaging scripts, but technical debt and drift (Bootstrap 4, Vue 2, pinned Python dependencies) are moderate.
This repository is a curated, static collection of real-world legal case studies documenting the consequences of web scraping, data theft, and automated tool misuse in China. It contains no executable code, data pipelines, or analytical notebooks; instead, it organizes news articles, court documents, and lawyer analyses into categorized 'forbidden zones' (禁区) covering illegal crawler services, personal data trading, copyright infringement, and service disruption. The sole entry point is the README, which lists 16+ cases with links to external sources. While not a software project, it serves as a valuable educational reference for developers and legal practitioners navigating Chinese data compliance law. The repository is fully reproducible as it consists of static markdown files with no dependencies or runtime requirements.
A monolithic Python application for crawling public data from the Chinese social network Weibo (Sina Weibo). It retrieves user profiles, weibo posts (including text, images, videos, live photos), comments, and reposts via the Weibo mobile API. Data can be exported to CSV, JSON, SQLite, MySQL, MongoDB, Markdown, or forwarded to a custom HTTP endpoint. It provides a CLI entry point for scheduled crawling (using the `schedule` library) and a Flask-based REST API for remote trigger and data retrieval. The codebase is heavily centralized—the core `Weibo` class (~3000 lines) handles fetching, parsing, downloading media, writing to multiple outputs, and anti‑ban throttling. A lightweight LLM analysis module can optionally enrich each weibo with sentiment, summary, and anomaly detection. The repository is community‑maintained with 4.6k stars, 402 open issues, and frequent commits as of mid‑2026. It is primarily an application but has a strong data‑collection character; the data outputs are suitable for subsequent research or archiving.
This repository implements a web scraper for the EastMoney stock forum (guba.eastmoney.com) using Selenium and MongoDB. It provides two main crawlers: PostCrawler scrapes post listings (title, views, comment count, URL, date, author) from paginated forum pages, and CommentCrawler scrapes individual post pages for comments (content, likes, date, sub-comments). The scraper uses a stealth.js script to evade bot detection, handles errors by restarting the WebDriver, and supports multi-threaded crawling for multiple stock symbols. Data is stored in MongoDB collections named post_<symbol> and comment_<symbol>. The code is a monolithic script with classes for crawling, parsing, and database operations. It is a functional but unremarkable scraper suitable for small-scale data collection, with basic error handling and no testing or dependency management.
This repository contains only documentation and configuration files (README, CONTRIBUTING, issue templates, .gitignore) with no actual source code present. The README describes SpiderSuite as a cross-platform web crawler for security professionals, but the codebase itself is empty in this snapshot. No functional analysis is possible. The repository appears to be a stub or the source code is hosted elsewhere. Given the absence of any executable or library code, this repository provides no reusable software assets.