Miller (mlr) is a command-line data processing tool for structured text formats including CSV, TSV, JSON, JSON Lines, YAML, DKVP, and others. It operates as a streaming pipeline: input records are read from files or stdin, processed through a chain of transformer verbs (e.g., cut, sort, join, stats1, put, filter), and written to output in any supported format. The tool includes a domain-specific language (DSL) for record-level transformations, with support for user-defined functions, subroutines, loops, and higher-order functions. Miller is designed for performance, using Go channels for concurrent streaming, batch processing, and a slab allocator for efficient memory use. The architecture is a monolithic CLI application written in Go with no external runtime dependencies. Key entry points include the main binary (cmd/mlr/main.go), the entrypoint dispatcher (pkg/entrypoint), and the terminal subcommands (help, repl, script, regtest, mcp). The codebase is organized into packages for input readers, output writers, transformers (verbs), DSL parsing and execution, CLI flag handling, and a core value type system (Mlrval). Miller also provides an MCP (Model Context Protocol) server for AI agent integration, shell completion scripts, and a REPL for interactive use. Miller is not a research project; it is a production-grade data munging tool comparable to awk, sed, cut, join, and sort but for name-indexed data. Its primary data flows involve reading heterogeneous records, applying transformations via verbs or DSL expressions, and writing results. The tool handles compressed files, in-place editing, and output redirection to files or pipes.
VisiData is a terminal-based interactive spreadsheet application for exploring, cleaning, restructuring, and analyzing tabular data. It provides a real-time curses interface for loading data from diverse sources (CSV, JSON, Excel, SQL databases, APIs, etc.), applying transformations via hundreds of keyboard-driven commands (sort, filter, pivot, melt, unfurl, etc.), and saving results. The architecture centers around a Sheet/Column abstraction with a plugin system for loaders, features, and specialized applications (vgit for Git, vdsql for SQL). Key strengths include its extensible command system, dynamic type inference, and reversible operations via undo/redo. While the core is cohesive, the extensive optional dependencies and experimental features introduce some maintenance complexity.
Vaex is a high-performance Python library for lazy out-of-core DataFrames designed to handle datasets larger than memory. It achieves efficiency through memory mapping, zero-copy policies, lazy evaluation, and virtual columns that avoid materializing intermediate results. The system supports HDF5, Apache Arrow, Parquet, CSV, and FITS formats, with accelerated operations via C++ extensions for critical paths like hash-based aggregations and string processing. The architecture is modular, consisting of a core package (vaex-core) and specialized extensions: vaex-viz (visualization), vaex-hdf5 (HDF5 enhancements), vaex-astro (astronomy tools), vaex-ml (machine learning integrations), vaex-jupyter (notebook support), vaex-server (remote access via WebSocket/REST), and vaex-graphql (GraphQL API). Key entry points include the `vaex` CLI tool, `vaex.open()` for data loading, and programmatic APIs for DataFrame operations, expressions, and visualizations. Data flows prioritize out-of-core processing: data remains on disk until needed, computations are lazily executed and cached via fingerprinting, and results are streamed to minimize memory footprint. The codebase demonstrates strong engineering practices with a plugin system for extensibility (expression/dataframe accessors), centralized settings configuration, and a task-based execution engine supporting parallelism and async operations. While primarily a general-purpose data analysis library, it is widely used in data research due to its efficiency with large datasets and domain-specific integrations (e.g., astronomy via vaex-astro). Recent commits indicate active maintenance, low technical drift through adoption of modern dependencies (Apache Arrow, FastAPI), and a comprehensive test suite covering core functionality and extensions.
Tablib is a Python library for working with tabular datasets in a format-agnostic way. It provides a core `Dataset` class that stores rows and optional headers, and a `Databook` class for collections of datasets. The library's primary functionality is converting between many structured data formats: CSV, TSV, JSON, YAML, Excel (XLS/XLSX), OpenDocument (ODS), dBase (DBF), HTML, Jira, LaTeX, reStructuredText, SQL INSERT statements, and pandas DataFrames. It also supports CLI table rendering via the `tabulate` package. The architecture is modular: each format is implemented as a class with `export_set`, `import_set`, `export_book`, `import_book`, and optionally `detect` methods. These are registered with a central `Registry` that also sets up dynamic property access on `Dataset` and `Databook` objects (e.g., `dataset.csv`, `dataset.json`). Lazy loading through importlib and string-based class references minimizes startup cost for optional formats like XLSX or YAML. The core `Dataset` object provides methods for row/column manipulation, sorting, filtering by tags, transposing, stacking, and applying formatters for custom cell transformations. Data flows are straightforward: users populate a `Dataset`, optionally set headers, then export to any registered format with a single `export()` call. Import reverses this process. Key entry points are `tablib.Dataset()`, `tablib.Databook()`, `tablib.import_set()`, `tablib.import_book()`, and `tablib.detect_format()`. The library has no hard runtime dependencies; all format-specific packages (openpyxl, xlrd, xlwt, odfpy, pyyaml, pandas, tabulate) are optional and requested via extras like `tablib[xlsx]` or `tablib[all]`. The vendored `dbfpy` module handles DBF format without external dependencies. Exception handling is structured with custom exception classes for invalid dimensions, unsupported formats, and missing headers.
OmniXAI is a comprehensive explainable AI (XAI) library providing a unified interface for generating explanations across multiple data types (tabular, vision, text, time series, ranking, vision-language) and ML models. It implements both model-agnostic (LIME, SHAP, L2X, MACE) and model-specific (Grad-CAM, Integrated Gradients, decision tree) explanation methods. The library features a modular architecture with separate modules for data handling, preprocessing, explanation generation, visualization (Dash-based dashboard), and deployment (BentoML integration). Key design patterns include base classes for explainers and explanations, factory patterns for explainer instantiation, and composable preprocessing pipelines. The system supports end-to-end workflows from data preprocessing to explanation visualization and model serving.
This repository is a curated awesome list and survey companion for diffusion models applied to time series, spatio-temporal, and tabular data. It contains no executable code; the sole content is a large README.md that organizes hundreds of papers into a taxonomy by task (prediction, generation, imputation, anomaly detection, classification/regression, causal inference, event prediction, foundation models) and by application domain (healthcare, sequential recommendation, weather, energy, finance, AIOps, environment, traffic, manufacturing, audio, biology, communication). Each entry links to the paper and, where available, official code repositories. It is the companion to an ACM Computing Surveys paper (DOI 10.1145/3783986) and includes citation information, a Google Sheets taxonomy, and an overview figure. The repository is actively maintained with recent commits and welcomes pull requests. As a research resource, it is highly reproducible: all content is in the markdown file, with no dependencies, datasets, or environment setup required. Although classified as data_research due to its research orientation, it is not a data pipeline or ML codebase. Its value is as a reference index and literature map for researchers; the README itself is the reusable asset.
This repository is a curated 'awesome list' for Federated Learning (FL) research, maintained as a Jekyll static site. It aggregates hundreds of linked GitHub repositories and DBLP search strategies. The core data asset is a machine-generated go.mod file that pseudo-pins the linked repositories as Go dependencies, produced by the add_dependent.py script which extracts GitHub URLs from README.md, queries the GitHub API for commit hashes, and writes a go.mod. However, this is not a functional Go module; it is a novelty lockfile. The dblp-search-keywords.md document provides a meticulous analysis of DBLP search syntax to discover FL papers without the 'federate' keyword. The repository has significant hybrid character: while it is primarily a content aggregation site (application-adjacent), its technical substance lies in the data exploration and search strategy documentation (research-oriented). The actual code is minimal and tightly coupled to the repository's specific structure, limiting reusability. The add_dependent.py script is a one-time scraper with hardcoded rate limiting and token placeholder, not a general-purpose tool.
This repository is a curated learning roadmap for machine learning, presented as a static GitHub Pages site. It contains no executable code, notebooks, or data pipelines. The sole content is a README.md file that organizes links to courses, books, frameworks, YouTube channels, blogs, datasets, and other resources for learning ML and deep learning. The site is built using Jekyll with the slate theme, configured via _config.yml. There are no application components, APIs, or data transformations. The repository serves as a reference guide rather than a software project. The architecture is trivial: a single-page static website. The entry point is the GitHub Pages URL, which renders the README. There are no data flows, no external dependencies beyond the Jekyll theme and GitHub Pages hosting. The repository has seen community engagement (529 stars, 81 forks) but no substantive code contributions. As a software project, this repository is not reusable in the traditional sense; its value lies in the curated list of resources. There are no components suitable for extraction. The maintenance appears active based on the inclusion of recent resources, but there is no code to maintain. Technical drift is low as there are no dependencies.