TensorFlow Datasets (TFDS) is a library providing ready-to-use public datasets for machine learning, primarily integrated with TensorFlow but also supporting JAX and PyTorch. It offers a standardized API (tfds.load, tfds.builder) for downloading, preparing, and loading datasets as tf.data.Dataset objects or numpy arrays. The architecture consists of core components (dataset builders, feature connectors, download management) and hundreds of dataset-specific implementations following a consistent pattern. Key entry points include the tfds CLI for dataset operations and the main tfds module for programmatic access. Data flows involve downloading raw data, generating TFRecord files via builders, and yielding examples through tf.data pipelines. External dependencies include TensorFlow, absl-py, numpy, pandas, and dataset-specific extras like apache-beam for large-scale processing or huggingface_hub for community datasets.
Masader is a metadata catalog for Arabic NLP datasets, providing structured information about over 100 datasets including their tasks, dialects, sources, volumes, and licensing. The repository combines a Jekyll-based static site with a modular JavaScript frontend to enable dataset discovery through search, filtering, statistics visualization, and a natural language chatbot interface. While the core asset is the dataset metadata (JSON files in /datasets/), the web interface and associated tools (annotation, migration, validation scripts) form an application layer that makes the catalog usable and maintainable. The system supports dataset exploration via faceted search, demographic charts, and embedding-based clustering, with a chatbot for conversational queries. Metadata is kept up-to-date through migration scripts that handle schema evolution and validation checks that ensure consistency with external sources like venue links and volume totals.
Diffgram is a full-stack data annotation platform designed for creating, managing, and reviewing annotations on images, video, audio, text, 3D point clouds, and geospatial data. The system is composed of multiple microservices: a Flask-based REST API (`default`), a background processing service (`walrus`) for media ingestion, export generation, and project migration, an event-driven action runner service (`eventhandlers`) that executes automated workflows (e.g., ML model inference, webhooks, exports), and a Vue 2 + Vuetify single-page application frontend. The backend uses PostgreSQL with SQLAlchemy ORM, RabbitMQ for async messaging, and supports multiple cloud storage backends (AWS S3, GCP, Azure, MinIO). Authentication supports OAuth2/Keycloak, API keys, and session cookies with fine-grained role-based permissions. The platform's core domain includes label schemas, attribute groups, instance templates, task/job management with review workflows, discussions/issues, and a custom query engine for filtering files by labels and attributes. It integrates with external AI services (HuggingFace, Vertex AI, Azure Text Analytics) and labeling vendors (Labelbox, Scale AI). The frontend provides rich annotation tools including 2D canvas (boxes, polygons, keypoints), 3D point cloud annotation with Three.js, audio waveform annotation, text token annotation, and geospatial tile annotation. A command pattern enables undo/redo for annotation actions. The codebase demonstrates solid engineering practices with clear separation of concerns, reusable patterns (strategy, observer, command, policy enforcer), and extensive test coverage (Cypress E2E, Jest unit tests, Python unittest). However, there are signs of technical drift: some outdated dependencies, mixed JavaScript/TypeScript in the frontend, and a few legacy modules marked as deprecated. The open-core model is evident from `OPENCORE - ADD` markers, suggesting proprietary extensions. Overall, Diffgram is a mature, actively maintained annotation platform suitable for teams building training data pipelines.
This repository is a curated list of public datasets, organized by topic (Agric Climate, etc.). It contains no source code, data pipelines, or software components. The entire repository is a README.rst file with links to external datasets, a license file, and a single zipped CSV (Datasets/titanic.csv.zip). The README states it is automatically generated from a separate project (apd-core), and direct modifications are discouraged. Many entries are marked with FIXME icons, indicating broken or outdated links. The repository serves as a reference index rather than a data engineering or analysis project.