Harmonized Landsat Sentinel-2 (HLS)

HLS Cloud usage exploited:

  • Ability to quickly setup clusters of servers
  • Access Cloud resident data
  • Vary system configurations as well as processors
  • Easily vary system storage needs and cluster sizes
  • Data harmonization allows users to channel data from different sources into a consistent, standardized, and comprehensive format for analysis.
  • HLS takes data from the joint NASA and USGS Landsat and the ESA Sentinel-2 satellites to create a harmonized dataset available for research.
  • HLS began in 2017 at NASA Ames on the NASA Earth Exchange (NEX) computer cluster.
  • The SMCE was used by the HLS effort to test, improve, tune, and perform quality control on the methods employed to harmonize the data.
  • SMCE was chosen due to:
    • The very high compute node storage demands (16TB EBS required on a cluster’s headnode), and up to 400TB of S3 storage.
    • Easy access to Sentinel-2 data sets, available from Google, but difficult to get direct from ESA.
    • Parallel clusters with 100’s of cpus to be deployed daily to process new daily data loads, but no need to pay for this compute between these cron job runs.
  • In early FY23 the HLS effort on the SMCE had completed processing 9 years of Landsat data and 7 years of Sentinel-2 data over 30% of the globe to verify and finalize the methods for deployment in production within MSFC to be stored in publicly accessible DACCS.