$ show electric-capital/open-dev-data
This repository is a curated, version-controlled taxonomy of cryptocurrency and blockchain ecosystems and their associated GitHub repositories. It is the dataset behind Electric Capital's developer activity reports. The primary artifact is a large collection of timestamped migration files (a custom DSL using commands like `repadd`, `ecoadd`, `ecocon`) that incrementally build a graph mapping ecosystems to repositories with tags. The dataset spans over a decade (2014–2026) and includes hundreds of thousands of repository entries across dozens of ecosystems, from major chains (Ethereum, Bitcoin, Stellar) to niche projects and hackathon submissions. The repository also provides a Python CLI tool (`open-dev-data`) for validating migrations, exporting the taxonomy as JSON Lines or Parquet, downloading the full dataset with resumable transfers and integrity checks, and exploring it locally via DuckDB and an interactive TUI (Harlequin). The codebase is well-structured, with clean abstractions for the data model, timestamp handling, and shell-like command parsing. However, the dataset itself suffers from significant noise—many entries are student hackathon repos, boilerplate forks, or duplicate URLs—which dilutes its signal for research. The curation is manual and not automatically deduplicated. Despite this, the taxonomy and tooling are highly reusable for any domain needing a version-controlled, hierarchically categorized registry of external projects.
The LLM identified 4 files worth highlighting for reuse:
Defines the core `Taxonomy` class that manages ecosystems, repositories, tags, and parent-child relationships via in-memory dictionaries. Provides methods to add/remove/rename entities and connect/disconnect ecosystems. This is the in-memory representation of the entire dataset and is reusable for any project that needs to maintain a hierarchical, graph-like registry of external resources. The design is clean and self-contained.
Implements asynchronous, resumable downloading of large Parquet datasets with exponential backoff, HTTP range requests, and blake3 checksum verification. The `download_file` function handles transient errors and provides rich progress bars. This pattern is directly reusable for any project that needs to reliably fetch large files from a remote manifest, with minimal dependencies (httpx, blake3).
Provides `has_valid_timestamp()` to validate ISO8601-like filenames (e.g., `2025-06-30T235959`) with leap-year and month-day correctness. The code is compact, well-tested, and easily extractable for any project that uses timestamped migration filenames. It demonstrates careful handling of edge cases (February 29th in non-leap years, month-day limits).
A custom shell-like tokenizer (`split()`) that handles quoted strings, escaped characters, and whitespace. Designed for parsing the migration DSL command arguments. It is a robust, dependency-free alternative to shlex for projects that need a lightweight, predictable tokenizer (e.g., for command parsing, configuration files). The implementation is concise and testable.