Pathway is a high-performance live data processing framework built on top of timely dataflow and differential dataflow, providing a Python API for defining streaming and batch data pipelines. It is designed for real-time ETL, event processing, AI/ML pipelines (including RAG), and incremental computation with strong consistency guarantees. The architecture consists of a Rust core engine that handles dataflow execution, scheduling, persistence, and connectors, while the Python layer provides a declarative API for table operations, UDFs, and integrations with external systems like Kafka, databases, vector stores, and cloud storage. Key entry points include the `pathway` CLI, the Python `pw` module, and the Rust engine library. Data flows through connectors (sources/sinks), transformation operators (joins, groupby, windowing, etc.), and optional persistence for fault tolerance. External dependencies include Kafka, S3, PostgreSQL, various vector databases, and LLM services.
cuDF is a GPU-accelerated DataFrame library developed by NVIDIA as part of the RAPIDS ecosystem. It provides a pandas-compatible Python API (`cudf`), a Polars-compatible execution engine (`cudf_polars`), a zero-code-change pandas accelerator (`cudf.pandas`), distributed processing via Dask (`dask_cudf`), a streaming distributed execution engine (`cudf_streaming`), Kafka integration, Java bindings, and a low-level C++ library (`libcudf`). In production, cuDF accelerates data manipulation, ETL, analytics, and ML preprocessing on NVIDIA GPUs by executing columnar operations in CUDA kernels while managing GPU memory through RMM. The monorepo contains over 5000 files spanning C++, Python, Java, Rust, and shell, with extensive unit tests, benchmarks, CI pipelines, and documentation. Key architectural patterns include consistent stream/memory-resource propagation via `cudf::detail` namespaces, a result cache for aggregation reuse, a fast-slow proxy pattern for transparent GPU/CPU fallback, and a streaming actor model for distributed pipelines.
Apache Superset is a production-grade business intelligence web application for data exploration and visualization. It provides a full-stack platform (Python Flask backend, React/TypeScript frontend) that connects to over 50 SQL databases, allowing users to create interactive dashboards, design charts via a plugin architecture, execute ad-hoc SQL queries in SQL Lab, and schedule email/slack reports with screenshots. The system manages metadata in a relational database (PostgreSQL/MySQL), uses Celery for background tasks (dashboard thumbnails, report execution), Redis for caching and async events, and exposes a REST API for programmatic access. Key architectural patterns include a command pattern for CRUD operations, DAO layer for data access, a rich chart plugin system with lazy-loaded rendering, and an embedded SDK for external dashboards. The codebase is actively maintained with frequent schema migrations, extensive tests, and an experimental MCP service for AI integration. While primarily an application, it includes data-research-adjacent features like semantic layers and pandas-based post-processing for chart data.
traffic is a Python library and CLI tool for processing, analyzing, and visualizing air traffic data. It ingests trajectory data from multiple sources including OpenSky Network (via REST/Trino), FlightRadar24, Eurocontrol (DDR, AIXM), FAA, and local ADS-B decoders. The core abstractions are `Flight` (single trajectory) and `Traffic` (collection) built on pandas DataFrames, with extensive methods for cleaning, resampling, filtering, geometric operations, navigation event detection (landing, go-around, holding patterns), performance estimation (fuel flow, emissions), and clustering. The library provides a declarative lazy evaluation pipeline (`LazyTraffic`) that chains operations before a single `.eval()` call, enabling efficient parallel processing. It also offers visualization backends for Matplotlib, Altair, Leaflet, and Plotly, and includes a Vue.js/Deck.gl plugin (Tangram) for real-time landing alignment in a web dashboard.
Feldera is a real-time streaming SQL analytics platform built on the DBSP (Differential Dataflow) incremental computation engine. It compiles SQL into efficient incremental circuits that process streaming data with sub-millisecond latency, supporting complex queries including joins, aggregations, window functions, and recursive queries. The system is deployed as a pipeline manager (Rust) with a REST API, a web console (Svelte), a Python SDK, a dbt adapter, and a CLI tool. It integrates with numerous data sources and sinks (Kafka, Postgres, Delta Lake, Iceberg, HTTP, NATS, Redis, etc.) and provides fault tolerance via checkpointing and storage backends. The architecture is polyglot: a Java-based SQL compiler (using Apache Calcite) translates SQL to Rust code that runs on the DBSP runtime; the pipeline manager orchestrates compilation, deployment, and lifecycle; the web console offers a full UI for pipeline management, monitoring, and profiling. The codebase also includes a Spark SQL translator (felderize) and extensive benchmark suites (Nexmark, LDBC). While primarily an application, it has strong research roots in incremental view maintenance and formal monotonicity analysis, reflected in the compiler's optimization passes and the DBSP runtime's operator design. Key reusable components include the adapter library (connector traits), the Feldera types crate (shared configs and errors), the common UI component library, and the Python client SDK. The project is actively maintained with frequent commits, comprehensive CI/CD, and a growing test suite covering SQL semantics across many data types and edge cases.
Frappe Insights is a full-stack business intelligence application enabling users to connect to data sources, build queries visually or via SQL, create visualizations, and assemble dashboards and workbooks. The decoupled architecture features a Vue.js 3 frontend communicating with a Python/Frappe backend through REST APIs. Key capabilities include a multi-database query builder using Ibis and SQLGlot, ECharts-powered visualizations, role-based access control, and DuckDB-based data warehousing for caching imported tables. The frontend leverages Vue composition API for reusable logic, while the backend uses Frappe's DocType system for extensibility. The project demonstrates solid engineering practices with modular API design, reusable components, and abstraction layers, though complexity arises from supporting numerous database types and deep Frappe integration.
Zui (formerly Brim) is an Electron-based desktop application for network security monitoring and data exploration. It provides a graphical interface to query and visualize data stored in a Zed/SuperDB lake, with a focus on Zeek and Suricata log analysis. The application supports importing packet captures (pcap), structured logs (Zeek TSV, NDJSON, CSV, Parquet), and offers a rich query editor (Monaco), histogram, table/inspector views, and export workflows. The codebase is a monorepo (Yarn workspaces, Nx) with packages for type definitions (`superdb-types`), a Node.js client (`superdb-node-client`), sample data, and an e2e test framework (`app-player`). The architecture follows a domain-driven pattern with handlers and operations, a Redux store with versioned state migrations, and a plugin system for extensibility. While the application is actively maintained and functionally rich, it exhibits some technical drift due to a mix of legacy and modern patterns, a large migration history, and tight coupling between core and plugins.
Diffgram is a full-stack data annotation platform designed for creating, managing, and reviewing annotations on images, video, audio, text, 3D point clouds, and geospatial data. The system is composed of multiple microservices: a Flask-based REST API (`default`), a background processing service (`walrus`) for media ingestion, export generation, and project migration, an event-driven action runner service (`eventhandlers`) that executes automated workflows (e.g., ML model inference, webhooks, exports), and a Vue 2 + Vuetify single-page application frontend. The backend uses PostgreSQL with SQLAlchemy ORM, RabbitMQ for async messaging, and supports multiple cloud storage backends (AWS S3, GCP, Azure, MinIO). Authentication supports OAuth2/Keycloak, API keys, and session cookies with fine-grained role-based permissions. The platform's core domain includes label schemas, attribute groups, instance templates, task/job management with review workflows, discussions/issues, and a custom query engine for filtering files by labels and attributes. It integrates with external AI services (HuggingFace, Vertex AI, Azure Text Analytics) and labeling vendors (Labelbox, Scale AI). The frontend provides rich annotation tools including 2D canvas (boxes, polygons, keypoints), 3D point cloud annotation with Three.js, audio waveform annotation, text token annotation, and geospatial tile annotation. A command pattern enables undo/redo for annotation actions. The codebase demonstrates solid engineering practices with clear separation of concerns, reusable patterns (strategy, observer, command, policy enforcer), and extensive test coverage (Cypress E2E, Jest unit tests, Python unittest). However, there are signs of technical drift: some outdated dependencies, mixed JavaScript/TypeScript in the frontend, and a few legacy modules marked as deprecated. The open-core model is evident from `OPENCORE - ADD` markers, suggesting proprietary extensions. Overall, Diffgram is a mature, actively maintained annotation platform suitable for teams building training data pipelines.
This repository is a curated skill library for data analysis workflows, designed to be consumed by AI agents (Claude Code, Codex). It contains 31 skills organized into 6 categories: data quality validation, documentation/knowledge management, data analysis/investigation, data storytelling/visualization, stakeholder communication, and workflow optimization. Each skill is defined by a SKILL.md file with YAML frontmatter, a step-by-step process, required inputs/outputs, and optional scripts, reference materials, and asset templates. The scripts are standalone Python CLIs that generate synthetic data for demo purposes and can be adapted to real datasets. The repo is not a deployable application or a data pipeline; it is a structured knowledge base and toolset for performing data analysis tasks programmatically. It covers common analytical patterns such as null counting, duplicate detection, cohort retention, funnel conversion, A/B testing, time series anomaly detection, metric calculation, and more. The codebase is well-organized and follows consistent conventions, but shows signs of staleness (future dates in README and LICENSE, no recent activity). Dependencies are not pinned, and no environment definitions are provided, limiting reproducibility without manual setup.