Easily vary system storage needs and cluster sizes
Data harmonization allows users to channel data from different sources into a consistent, standardized, and comprehensive format for analysis.
HLS takes data from the joint NASA and USGS Landsat and the ESA Sentinel-2 satellites to create a harmonized dataset available for research.
HLS began in 2017 at NASA Ames on the NASA Earth Exchange (NEX) computer cluster.
The SMCE was used by the HLS effort to test, improve, tune, and perform quality control on the methods employed to harmonize the data.
SMCE was chosen due to:
The very high compute node storage demands (16TB EBS required on a cluster’s headnode), and up to 400TB of S3 storage.
Easy access to Sentinel-2 data sets, available from Google, but difficult to get direct from ESA.
Parallel clusters with 100’s of cpus to be deployed daily to process new daily data loads, but no need to pay for this compute between these cron job runs.
In early FY23 the HLS effort on the SMCE had completed processing 9 years of Landsat data and 7 years of Sentinel-2 data over 30% of the globe to verify and finalize the methods for deployment in production within MSFC to be stored in publicly accessible DACCS.
