TensorFlow Datasets (TFDS) is a library providing ready-to-use public datasets for machine learning, primarily integrated with TensorFlow but also supporting JAX and PyTorch. It offers a standardized API (tfds.load, tfds.builder) for downloading, preparing, and loading datasets as tf.data.Dataset objects or numpy arrays. The architecture consists of core components (dataset builders, feature connectors, download management) and hundreds of dataset-specific implementations following a consistent pattern. Key entry points include the tfds CLI for dataset operations and the main tfds module for programmatic access. Data flows involve downloading raw data, generating TFRecord files via builders, and yielding examples through tf.data pipelines. External dependencies include TensorFlow, absl-py, numpy, pandas, and dataset-specific extras like apache-beam for large-scale processing or huggingface_hub for community datasets.
This repository provides a JSON dataset of world cities and their administrative divisions (levels 1 and 2) sourced from the GeoNames Gazetteer. The data includes city names, coordinates, country codes, and administrative codes for over 169,000 cities. The repository features automated monthly updates via GitHub Actions that download raw GeoNames data, convert it to JSON format, and validate the output. The data structure enables direct use in geolocation applications, administrative boundary mapping, and demographic analysis without additional processing.
This repository is a community-curated directory of public APIs, organized by category in a single README.md file. Each entry includes the API name, description, authentication type, HTTPS support, and CORS status. The primary data artifact is the markdown file itself, which is manually maintained through pull requests. Supporting Python scripts validate the markdown format (alphabetical order, column padding, description length) and check that all listed API links are reachable, with special handling for Cloudflare-protected endpoints. The validation scripts are well-structured, parameterized, and include a comprehensive test suite, making them reusable for any similar markdown-based table validation. The repository is actively maintained with frequent commits and CI workflows, though its Python dependencies are slightly outdated (certifi 2021, requests 2.27.1). The core value is the curated dataset, not a software application.
PLINDER is a comprehensive protein-ligand interaction dataset and evaluation resource, providing over 400,000 annotated systems with 750+ annotation columns derived from the PDB. The repository contains a full data generation pipeline (Metaflow-based, configurable via YAML) that downloads, annotates, scores, clusters, and splits structures into train/val/test sets with leakage control. It also includes a Python API (PlinderSystem, PlinderDataset) for accessing the dataset, querying similarity scores, and loading structures, as well as an evaluation harness (ModelScores, StratifiedTestSet) for benchmarking docking algorithms using metrics like lDDT-PLI, DockQ, and posebusters checks. The codebase is hybrid: primarily a data research project with strong software engineering practices (Docker, CI/CD, extensive tests, config-driven pipelines), but the core value is the curated dataset and the tools to generate and evaluate it.
This repository is the canonical source data for the ZenginCode ecosystem, providing a curated, machine-readable dataset of Japanese bank codes and branch codes. It contains structured branch information (code, name, kana, hira, roma) for thousands of financial institutions across Japan, stored in both JSON and YAML formats under data/branches/. The data is automatically updated daily via a GitHub Actions workflow that downloads the official Zengin bank branch list from an external source, converts it from Shift_JIS CSV to UTF-8, and generates per-bank branch files. The repository also includes a Ruby CLI tool (bin/zengin) that orchestrates the download, conversion, and commit process, making the pipeline reproducible. While primarily a data repository, it includes a well-structured ETL pipeline that can serve as a template for similar data curation projects. The dataset is consumed by language-specific packages (Ruby, JavaScript, Python) to provide bank branch lookup functionality in financial applications.
This repository provides the Bench2Drive benchmark for closed-loop end-to-end autonomous driving, accepted at NeurIPS 2024 Datasets and Benchmarks Track. It includes a large-scale multi-modal dataset (up to 7.3TB) collected in the CARLA simulator, an evaluation suite with metrics like Driving Score and Success Rate, and baseline results for over 10 methods. The codebase is a hybrid of research data pipeline and application: the dataset generation and evaluation scripts form the core data_research component, while the B2DVisualize Flask app, scenario runner, and leaderboard evaluator provide application-like interfaces for interaction and reproducibility. Data lineage: raw simulation data (sensor recordings, ground truth annotations) is collected via data_collect.py, transformed into compressed archives with JSON manifests, and evaluated using the leaderboard pipeline that computes multiple ability metrics. The repository answers research questions about the effectiveness of end-to-end driving policies under diverse urban scenarios.
The CircuitNet repository provides an open-source dataset and associated code for machine learning in Electronic Design Automation (EDA). It focuses on four key EDA tasks: congestion prediction, design rule check (DRC) prediction, IR drop prediction, and net delay prediction. The repository includes scripts for extracting features from LEF/DEF files (via Innovus output), training and evaluating deep learning models (CNNs and GNNs) for these tasks, and utilities for data loading and model training. The dataset is distributed in multiple versions (CircuitNet-N28, N14, N45) and consists of preprocessed features and labels stored as NumPy arrays. The codebase is organized into modular components: feature extraction, task-specific model directories, and shared utilities. While the repository enables reproduction of published results, it relies on specific tool versions (Innovus) and has hardcoded paths, which may require adaptation for reuse in different environments.
The Multimodal Universe is a large-scale astronomical data repository that aggregates ~30 surveys (Legacy Survey, HSC, JWST, Gaia, SDSS, DESI, TESS, etc.) into a unified, HEALPix-partitioned HDF5 format, exposed through Hugging Face datasets. It covers multiple modalities: images (multi-band), spectra (1D), time series (light curves), and tabular catalogs. The repository provides dataset builder scripts (GeneratorBasedBuilder subclasses) per survey, cross-matching utilities, and benchmark training code (PyTorch Lightning) for downstream tasks like photometric redshift estimation and galaxy property regression. While primarily a data research project, it also includes application-like components such as baseline classifiers (Informer for PLAsTiCC) and experimental benchmark pipelines. The data is designed for training foundation models in astronomy, with streaming support and standardized column naming. Notable patterns include HEALPix-based spatial indexing, parallel processing in build scripts, and a consistent Hugging Face dataset interface across all surveys. The codebase is well-structured for its purpose, though many scripts are tightly coupled to specific surveys and external data sources, limiting direct reuse without adaptation.
This repository is a curated list (``awesome list'' style) of high-quality datasets and tools for LLM fine-tuning, post-training, and alignment. It is not a software application, data pipeline, or research codebase; it contains no executable code, notebooks, or data files. The primary artifact is a README.md that categorizes and links to over 70 external datasets (instruction, preference, multilingual, agent, etc.) and tools for data scraping, filtering, generation, and exploration. The repo serves as a reference guide for practitioners seeking datasets, with a focus on accuracy, diversity, and complexity. It is maintained by Maxime Labonne and has a large number of stars and forks, indicating community value. The repository has no code to analyze, test, or reproduce; its value lies in the curation and annotation of external resources.
This repository functions as a curated "Awesome List" for Japanese Natural Language Processing (NLP), aggregating metadata from GitHub and Hugging Face. It includes a Claude Code plugin that allows searching the 1,197-resource dataset directly within the AI interface. The data lineage involves manual curation and automated scraping of external repositories, stored in `resources.json` and Markdown documentation. Critical risks include significant link rot (many "repo not found" entries), a suspicious future commit date (2026-07-20), and high technical drift due to the rapidly changing NLP landscape. While valuable for discovery, the static nature of the data and lack of environment definitions make it unsuitable for production reliance without verification.
This repository is a static dataset of LeetCode coding problems curated by company and time period. It contains CSV files organized into directories named after hundreds of companies (e.g., Google, Amazon, Meta, Apple, Goldman Sachs), each with subfiles for different time ranges: 'Thirty Days', 'Three Months', 'Six Months', 'More Than Six Months', and 'All'. Each CSV row includes problem difficulty, title, a frequency metric (normalized so the top problem equals 100), acceptance rate, link, and topic tags. The data is a snapshot as of June 2025, derived from aggregated user reports or scraping. There is no executable code, no transformation scripts, and no automated update mechanism. Many CSV files are empty or contain incomplete data, indicating gaps in coverage. The repository serves as a reference for interview preparation but lacks reproducibility and documentation of data provenance.
This repository is a research artifact from the NL2Bash project (LREC 2018), providing a dataset and neural semantic parser for translating natural language descriptions into Bash one-liners. The core contributions include a parallel corpus of ~10k NL–Bash pairs, TensorFlow-based sequence-to-sequence models with attention and copy mechanisms, a custom Bash command parser (bashlint) that normalizes commands using a grammar extracted from man pages, and evaluation tools including tree edit distance and manual judgment pipelines. The codebase is primarily a research prototype with pre-trained model checkpoints, data preprocessing scripts, and evaluation files. While the dataset and parser are valuable, the dependency on TensorFlow 1.x, JavaCC, and Python 2/3 compatibility indicates significant drift. The repository is no longer actively maintained, but the documented methodology and reusable components (e.g., tree edit distance, statistical spell checker) remain relevant for similar NL-to-code tasks.
This repository provides two LiDAR degeneracy datasets (tunnel and fog) in ROS bag format, hosted on Hugging Face. It includes detailed metadata about the sensors (IMU, LiDAR, radar), their extrinsics, topics, and rates. The datasets are intended for research in LiDAR-radar-inertial fusion for challenging environments. The repository contains no code for processing the data, only documentation and links to the raw data.
This repository is a curated 'awesome' list of biomechanical resources. It contains no executable code, notebooks, or data pipelines. The entire content is a single README.md file with hundreds of categorized links to learning materials, datasets, software tools, books, societies, and other resources relevant to biomechanics research and education. It functions as a community-maintained directory rather than a software project or research pipeline. While the list is well-organized and frequently updated, there are no computational components to analyze or reuse.
This repository is a curated collection of five large-scale Chinese NLP datasets (Wikipedia articles, news, QA pairs, community Q&A, and translation pairs) with descriptions and download links. It functions as a data catalog rather than a software project: there is no executable code, no ETL scripts, and no pipelines. The datasets are hosted externally (Google Drive, Baidu) and require manual download. The repository provides clear formatting examples and suggested use cases, but lacks any automation or reproducibility support such as download scripts, environment definitions, or data processing code. It is a valuable reference for researchers needing Chinese text corpora, but it is not a reusable data engineering artifact.