$ show metadriverse/scenarionet
ScenarioNet is a data-centric toolkit for managing and converting autonomous driving datasets into a unified scenario format. It provides converters for Waymo, nuScenes, nuPlan, Argoverse 2, VoD, and procedural generation (PG) datasets, transforming raw data into a common ScenarioDescription structure compatible with the MetaDrive simulator. The repository includes dataset management operations (merge, split, filter, copy), verification tools (existence and simulation checks), and utilities for handling scenario metadata. While it contains training and evaluation scripts for machine learning models, its primary focus is on data preparation and scenario engineering for autonomous driving research. The data lineage involves ingesting raw datasets via dataset-specific toolkits, extracting scenario elements (object tracks, map features, traffic signals, timestamps), and serializing them into pickle-based dataset structures with mapping and summary files. These outputs enable downstream tasks such as multi-dataset aggregation, scenario filtering for ML training, simulation-based validation, and dataset splitting. The toolkit addresses research questions around dataset scalability, scenario diversity, and reproducible evaluation of autonomous driving systems by providing a standardized interface across heterogeneous data sources. Key reusable components include the modular converter utility for multiprocessing data transformation, generic dataset operations (merge/split/copy) that operate on the standardized scenario format, and a verification framework with parallel processing and error reporting. These components demonstrate solid data engineering practices but are tempered by dependency management issues (commented-out packages requiring manual installation), hardcoded paths in training scripts, and reliance on external SDKs that may introduce version drift.
The LLM identified 3 files worth highlighting for reuse:
Provides write_to_directory for efficient multiprocessing dataset conversion and helper functions (e.g., unit conversions). Isolated from dataset-specific logic, making it extractable for any bulk data conversion task requiring parallel processing.
Implements merge_database, split_database, and copy_database functions that work on the standardized scenario format (dataset_mapping.pkl and dataset_summary.pkl). These generic operations can be reused with any dataset adhering to the same structure, featuring options for filtering and sequential/random splitting.
Contains verify_database for validating scenario datasets via multiprocessing, checking loadability and simulation feasibility. Includes error reporting and the ability to generate new datasets from failures, showcasing a reusable pattern for large-scale data validation with robust fault tolerance.