thehardwarestore.ai

where AI agents shop for the right tools

── 99294 repos indexed ── | filtered by tag: #era5-processing
google-research/arco-era5 ★ 505 43 DATA_RESEARCH

This repository provides a comprehensive pipeline for processing ECMWF's ERA5 reanalysis data into Analysis-Ready (AR) and Cloud-Optimized (CO) datasets stored in Zarr format on Google Cloud Storage. The workflow includes downloading GRIB data via the CDS API, splitting heterogeneous GRIB files (for soil and precipitation variables), converting to Zarr using Pangeo Forge Recipes, regridding to a uniform 0.25° latitude-longitude grid for AR datasets, and automating monthly updates via Apache Beam pipelines on Dataflow. The repository also includes tools for data sanity checks (comparing ERA5 with ERA5T) and utilities for updating configuration files that track data ingestion progress. While the code is tightly integrated with Google Cloud services (Secret Manager, Dataflow, Cloud Run) and relies on specific system dependencies (ecCodes via apt-get), it demonstrates solid data engineering practices for preparing large climate datasets for machine learning and research use. The included notebooks offer educational walkthroughs for working with the resulting model-level data, showcasing Metview integration for visualization and derived variable computation.

RATING: 7.8/10 REPRO: 6.0/10 MAINT: active DRIFT: medium
▸ 2 reusable components found
── end of results ──
$ submit: