$ show google-research/arco-era5
This repository provides a comprehensive pipeline for processing ECMWF's ERA5 reanalysis data into Analysis-Ready (AR) and Cloud-Optimized (CO) datasets stored in Zarr format on Google Cloud Storage. The workflow includes downloading GRIB data via the CDS API, splitting heterogeneous GRIB files (for soil and precipitation variables), converting to Zarr using Pangeo Forge Recipes, regridding to a uniform 0.25° latitude-longitude grid for AR datasets, and automating monthly updates via Apache Beam pipelines on Dataflow. The repository also includes tools for data sanity checks (comparing ERA5 with ERA5T) and utilities for updating configuration files that track data ingestion progress. While the code is tightly integrated with Google Cloud services (Secret Manager, Dataflow, Cloud Run) and relies on specific system dependencies (ecCodes via apt-get), it demonstrates solid data engineering practices for preparing large climate datasets for machine learning and research use. The included notebooks offer educational walkthroughs for working with the resulting model-level data, showcasing Metview integration for visualization and derived variable computation.
The LLM identified 2 files worth highlighting for reuse:
Contains a function to split GRIB files by variable using `weather-sp`, which is essential for handling heterogeneous GRIB files (like soil and precipitation data) that store multiple variables with different grids in a single file. This function is self-contained (only depends on a small utils module) and solves a recurring problem in meteorological data processing.
Implements a secure, configuration-driven download of ERA5 data from the Copernicus CDS API, using Google Secret Manager to manage API keys. It abstracts the download process, handles errors, and updates local file tracking. This pattern can be adapted for other data sources that require API keys and configuration-based downloads.