TrendRadar is a configurable news aggregation and trend analysis system that crawls hot lists from 11+ Chinese platforms (Zhihu, Weibo, Douyin, Bilibili, etc.) and RSS/Atom feeds, filters content via keyword DSL or AI-based interest matching, performs AI-powered analysis (sentiment, controversy, topic trends), generates rich HTML reports, and pushes notifications to 9+ channels (Feishu, DingTalk, Telegram, Email, ntfy, Bark, Slack, WeCom, generic webhook). It is primarily an application (with CLI entry points, scheduling via timeline.yaml, Docker/GitHub Actions deployment) but incorporates data-pipeline patterns such as multi-stage ETL (crawl โ store โ filter โ analyze โ report), SQLite storage with optional S3 remote sync, and AI batch processing. The architecture is a modular monolith: the core `trendradar` package handles all orchestration, while a separate `mcp_server` package exposes 20+ tools via Model Context Protocol (MCP) for LLM integration. The system is heavily config-driven (YAML and text files) and includes a standalone web-based config editor. External dependencies include the third-party newsnow API (for hot lists), LiteLLM (for AI model abstraction), and various notification service APIs. The codebase demonstrates good craftsmanship with clean abstractions (storage backends, notification dispatcher, AI client), though some duplication exists between the core and MCP notification logic, and the extensive configuration surface requires careful maintenance.
This repository is a curated collection of 177+ domain-specific scientific 'skills' for AI agents, following the Agent Skills standard. Each skill is a self-contained module providing a CLI tool, documentation, and tests for a specific scientific or technical taskโranging from single-cell RNA-seq analysis (scanpy) and molecular docking (diffdock) to clinical report validation and ISO standards readiness. The architecture is a monorepo of independent skills, each with its own scripts/, references/, assets/, and tests/. Skills are designed to be safe, bounded, and reproducible: they validate inputs before execution, enforce resource limits, avoid network access unless explicitly required, and produce structured JSON reports. The repository also includes a shared test infrastructure (skill_contract) and a meta-skill (autoskill) that can automatically propose new skills from screen activity. While primarily an application library for AI agents, it has strong data/research characteristics due to its heavy use of scientific data formats and validation of research artifacts.
Python-100-Days (Python 100ๅคฉไปๆฐๆๅฐๅคงๅธ) is one of the most widely referenced Chinese-language Python learning curricula on GitHub (~186k stars). Despite its scale, it is fundamentally an educational artifact rather than a deployable software system: the dominant content is a 100-day sequence of Markdown tutorials (Day01-100) augmented with teaching-oriented code examples, SQL schema scripts, and Jupyter notebooks. The closest software classification is data_research, driven by the substantial data-analysis track โ Day66-80 covers NumPy, pandas, and Matplotlib against real bundled datasets (scraped BOSS็ด่ job postings across nine Chinese cities, 2020 sales data, 2023 Beijing hukou data), and Day81-90 covers scikit-learn machine learning with classic datasets (iris, Titanic, MPG, Boston housing). The repo's character is hybrid: it also carries a Django web-development track (Day46-60), a web-scraping track with Scrapy/Selenium (Day61-65), and database/DevOps chapters (Day36-45, Day91-100), but none of these constitute a runnable production application โ they are teaching material. Ground truth from the code: the only real end-to-end 'data pipeline' is the job-market dataset pipeline. Raw CSVs under Day66-80/code/res/jobs/ contain scraped listings from zhipin.com with a consistent schema (company_name, uri, salary, site, year, edu, job_name, city, pos_count); the notebook workflow produces a cleaned artifact (cleaned_jobs-checkpoint.csv) with salary ranges parsed into salary_lower/salary_upper bounds. This demonstrates a reusable salary-parsing and job-market normalization pattern. The machine-learning chapters are textbook-style reproductions of classical algorithms for pedagogy, not original research, and the scraping data is a timestamp-less snapshot. Maintenance is active: the author continues to revise the curriculum (a 2025 version reset of the introductory series) and the last commit is recent. Technical drift is medium โ older chapters reference Django 2.2, Vue 2.6.10, ECharts 4.2.1, and Python 3.7-era idioms, while some third-party API examples (TianAPI, chaojiying captcha OCR, SMS gateways) depend on external services that may have changed. Reproducibility is partial: dataset files needed by notebooks are checked in, but the main curriculum has no pinned dependency manifest (only the opencourse subproject has requirements.txt), no conda environment file, and the scraped data has no crawl timestamp.
BettaFish (ๅพฎ่) is a multi-agent public opinion analysis system that automates the end-to-end process of gathering, analyzing, and reporting on public sentiment across 30+ Chinese social media platforms. The system is built around four specialized agents: QueryEngine (news breadth search via Tavily), MediaEngine (multimodal search via Bocha), InsightEngine (database mining via SQLAlchemy), and ReportEngine (template-based report generation with intermediate representation and HTML/PDF/Markdown rendering). A ForumEngine coordinates agent collaboration through log monitoring and LLM-generated moderator comments. The architecture is a monolithic Flask application that manages multiple Streamlit sub-applications and CLI tools, with a separate MindSpider crawler subsystem for data collection. While primarily an application, it includes significant data/research components such as sentiment analysis model training scripts (BERT, GPT-2, Qwen3, traditional ML), a Weibo sentiment dataset, and data pipeline scripts. The system relies heavily on external LLM APIs (OpenAI-compatible), search APIs (Tavily, Bocha, Anspire), and system dependencies (Playwright, WeasyPrint). The codebase demonstrates solid engineering with reusable patterns like robust JSON parsing, chart validation/repair, and a hardened GitHub Actions star-history script, but suffers from significant code duplication across engines and tight coupling to external APIs.
DataEase is a self-hosted, open-source business intelligence (BI) platform that enables users to connect to diverse data sources, build datasets, create interactive charts and dashboards, and share insights via links or embedded widgets. The codebase is a full-stack monolith with a Java Spring Boot backend (JPA, Calcite SQL parser, Quartz scheduling) and a Vue 3 frontend (AntV G2/G2Plot/L7 for charts, UniverJS for spreadsheets). It supports a wide range of database connectors (MySQL, PostgreSQL, Oracle, ClickHouse, StarRocks, Elasticsearch, MongoDB, etc.), dataset types (SQL, union, custom fields), chart types (bar, line, pie, map, table, indicator, etc.), and enterprise features like row/column-level permissions, scheduled sync, report delivery, and multi-factor authentication. The architecture includes an SDK layer for extensions (datasource providers, sync providers, chart plugins) and a separate Xpack module for commercial features. The platform is deployed via Docker Compose with APISIX gateway. The codebase is actively maintained (24k+ stars, recent commits) but exhibits some technical drift due to deep inheritance hierarchies in chart handlers, code duplication across engine providers, and tight coupling to specific library versions (AntV G2, UniverJS). Overall, it is a solid, production-grade BI tool with good separation of concerns but with room for refactoring in certain areas.
Nemesis is an offensive security data processing platform by SpecterOps that ingests, enriches, analyzes, and alerts on files collected during red team and DFIR operations. The system is architected as a suite of microservices orchestrated via Dapr (pub/sub, workflows, state management), with services implemented in Python (FastAPI), Go (Titus secret scanner), and C# (ILSpy .NET decompiler). Key data flows: files are ingested via a CLI (supporting Cobalt Strike, Mythic, Outflank connectors) or REST API, then processed through a Dapr workflow that performs basic analysis (magic, hashes), runs pluggable enrichment modules (PE, .NET, Chromium/DPAPI decryption, YARA, SQLite, containers), invokes the Titus secret scanner, and finally publishes findings and alerts. A React frontend with Hasura GraphQL provides a dashboard for file browsing, enrichment status, YARA management, LLM-based triage agents, and reporting. The platform is deployment-agnostic with Docker Compose and Helm charts for k8s. Notable technical depth includes a standalone DPAPI decryption library (nemesis_dpapi) with multiple storage backends and write-once semantics, a rule-based file linking engine, and an AI agent framework using pydantic-ai and LiteLLM.
A full-stack personal running activity dashboard that syncs workout data from over a dozen platforms (Strava, Garmin, Nike, Keep, Coros, etc.) via Python scripts, stores it in SQLite, and renders interactive web visualizations (React + Mapbox) and SVG posters (grid, heatmap, year summaries). It also includes a terminal UI (Textual) and an optional AI-powered share image generator. The architecture is a hybrid monolith: a shared SQLite database backs both the frontend (exported as JSON) and SVG generation. The frontend is themeable with two built-in themes (classic, dashboard). While the sync scripts are tightly coupled to each provider's API, the SVG poster drawer library and the terminal UI's data aggregation layer are well-isolated and potentially reusable. The project is actively maintained with frequent commits, CI/CD pipelines, and Docker support, though it carries moderate technical drift due to many unpinned dependencies and reliance on third-party APIs that may change.
KC3Kai is a mature Chrome extension for the browser game Kantai Collection (KanColle), providing an enhanced game viewer, data analysis, and automation tools. It operates as a DevTools panel and content script, intercepting game API calls to display real-time fleet status, battle predictions, quest tracking, resource graphs, and expedition scoring. The architecture is monolithic with global singletons (KC3Master, KC3SortieManager, PlayerManager) and a tab-based Strategy Room UI for historical data analysis (ship drops, event stats, PvP replays, Akashi improvement planner). It integrates with community databases (TsunDB, PoiDB, KCRDB) for data submission and uses IndexedDB for persistent storage of sorties, battles, and encounters. While primarily an application, it contains significant data analysis components such as battle prediction algorithms, anti-air cut-in calculations, and quest synchronization logic, making it a hybrid tool with strong data engineering aspects.
SenseNova-Skills is a composable collection of AI agent skills for office automation, built around the SenseNova model family. It provides production-grade capabilities for image generation and recognition, presentation (PPT) creation, Excel data analysis, deep research with structured evidence, and multi-source search (academic, social, code, finance). The skills are designed to run in Agent Skills-compatible runtimes (OpenClaw, hermes-agent) and are bundled into the Raccoon product for enterprise use. The architecture is modular: each skill is a self-contained directory with a SKILL.md manifest, scripts, prompts, and schemas. The deep research skill implements a multi-agent pipeline with formal JSON schemas and validators. The PPT skill has a multi-stage pipeline from user input to HTML-to-PPTX conversion via Playwright. The codebase demonstrates solid engineering with reusable components like a shared LLM/VLM client, export pipeline libraries, and validation infrastructure. However, it suffers from duplicated utility code across skills, complex environment variable fallback chains, hardcoded paths, and dependency on external APIs and headless browsers. The repo is actively maintained with biweekly reports and frequent commits.
Kanaries/Rath is an open-source augmented analytics BI platform that automates exploratory data analysis, causal discovery, and dashboard creation. It is a full-stack application with a React/TypeScript frontend (Vite, MobX, shadcn/ui) and multiple Python microservices (causal inference, narrative insight generation, database connectors). The system ingests data from files, databases (PostgreSQL, ClickHouse, MySQL, etc.), and REST APIs, then provides automated visualization (mega-automation), semi-automated exploration, causal analysis (PC, FCI, GES, XLearner), and interactive dashboard building. It also includes custom Vega renderers (WebGL, Painter) for high-performance chart rendering and an Electron desktop wrapper. While primarily an application, it contains significant data research components such as causal discovery algorithms, statistical insight detection, and text pattern extraction, making it a hybrid tool for both data exploration and advanced analytics.
This repository powers the personal finance blog 'OfDollarsAndData.com'. It is primarily a data research collection, consisting of hundreds of R scripts that import, clean, and analyze financial and economic datasets to produce publication-ready charts, tables, and interactive calculators for blog posts. The data lineage is clear: raw data is downloaded or read from external sources (Shiller S&P 500, FRED, BLS, Survey of Consumer Finances, YCharts, etc.) in import/ scripts, transformed and saved as RDS files in build/ scripts, and then consumed by analysis/ scripts that generate plots with a consistent ggplot theme. The repository also includes a webdev folder containing a WordPress theme with custom widgets and calculator embedding logic. While the codebase is not a packaged application and lacks testing or CI, the analysis is thorough and well-structured for its purpose. The hybrid nature includes both data research R scripts and web development PHP/JS code, but the dominant character is data research.
This repository is a curated skill library for data analysis workflows, designed to be consumed by AI agents (Claude Code, Codex). It contains 31 skills organized into 6 categories: data quality validation, documentation/knowledge management, data analysis/investigation, data storytelling/visualization, stakeholder communication, and workflow optimization. Each skill is defined by a SKILL.md file with YAML frontmatter, a step-by-step process, required inputs/outputs, and optional scripts, reference materials, and asset templates. The scripts are standalone Python CLIs that generate synthetic data for demo purposes and can be adapted to real datasets. The repo is not a deployable application or a data pipeline; it is a structured knowledge base and toolset for performing data analysis tasks programmatically. It covers common analytical patterns such as null counting, duplicate detection, cohort retention, funnel conversion, A/B testing, time series anomaly detection, metric calculation, and more. The codebase is well-organized and follows consistent conventions, but shows signs of staleness (future dates in README and LICENSE, no recent activity). Dependencies are not pinned, and no environment definitions are provided, limiting reproducibility without manual setup.
This repository is a 30-day Python tutorial that originated as an application-focused learning project but has evolved into a hybrid data_research resource. The core is a structured curriculum covering Python fundamentals through 30 days of markdown documentation and accompanying Python scripts. However, the repository's most substantial and unique content lies in its data folder, which contains rich datasets including presidential speeches (Obama, Trump, Michelle Obama, Melina Trump), Fortune 500 company data, Hacker News posts, country statistics, email exchanges, and weight-height CSV data. The data_research character emerges from the later days which cover web scraping (with actual scraped JSON output from Boston University), Pandas data analysis, NumPy statistics, MongoDB database operations, and API consumption. The repository includes Flask web applications for text analysis, demonstrating practical data processing pipelines. While the tutorial content is well-structured and professionally presented across multiple languages (English, German, Spanish, Chinese, Portuguese, Persian, Korean, Ukrainian), the actual executable Python code is minimal compared to the markdown documentation. The data assets are the most valuable component - they're real, substantial datasets that could be used for NLP analysis, statistical exercises, and data visualization projects. However, the repository lacks proper dependency management beyond a basic requirements.txt for the Flask app, has no environment configuration, and the data sources are not systematically documented with metadata or provenance information.
This repository is a curated list of resources for practicing data science with Python, organized in the style of 'awesome' lists. It contains links to libraries, tutorials, blog posts, talks, code snippets, and other resources across various data science domains including core libraries, machine learning, NLP, computer vision, and more. The repository serves as a reference guide for data scientists and practitioners rather than containing executable code or performing data processing tasks.
This repository is a Chinese translation (2nd edition) of Wes McKinney's "Python for Data Analysis" book, presented as markdown chapter files. It is an educational resource, not a software application or a full data pipeline. The content covers pandas, NumPy, Matplotlib, and other data analysis libraries with embedded code examples for learning purposes. It is not a project that can be executed directly; there are no Python scripts, notebooks, or data files included. The code examples are based on older library versions (circa 2017) and use deprecated APIs such as `ix` and `read_table`. External datasets referenced (e.g., tips.csv, MovieLens) are not bundled, making reproduction impossible without manual setup. The repository has not been actively maintained, and its technical drift is high due to ecosystem changes. It is best used as a reference or study guide, not as a reusable codebase.