SST Validation Description

The SeaBASS sea surface temperature (SST) validation system is designed to provide ground-truth of satellite-borne measurements via comparisons with coincident in situ temperature measurements. Successful match-ups are compiled into one global file per date for each different sensor and made available via a simple web-based search engine.

 

This article explains how to search for and download results, how to work with validation files, and provides background information describing the steps involved in creating validation match-ups.



Table of Contents


Obtaining and Working with Data

All global match-up results are compiled into daily data files, one file per date, per each different satellite sensor. Obtaining these SST validation files involves a two-step process of searching then downloading. 1) Search for a list of URLs of the files that meet your sensor and date criteria using either the web-browser graphical interface or the API search method. 2) Run a script in order to retrieve and download all of the data files in your list. You can then read through multiple files in order to combine data from multiple measurement dates.

 

You must select at least one satellite sensor in your search query. Optionally, you may adjust the date ranges when measurements occurred (“measured between”), and/or the date ranges when results files were most recently created or modified (“archived between”). Archived dates are modified whenever new or historical in situ datasets are processed (which adds or modifies existing daily files), or when satellite data are reprocessed.

 

 

Web browser search method

 

  • (Required) Select at least one satellite sensor checkbox
  • (Optional) Use the date range filters to filter the results
  • Press the “Search” button or the “Perform search in new tab” button. Individual data files may be downloaded one at a time, or else follow the How To Download instructions further below to download in bulk

 

API search method

 

Get-requests may be crafted using the following parameters. One or more "sensor" entries are required:

 

sensor (MODIS-Aqua, MODIS-Terra, VIIRS-SNPP)
startdate (earliest measurement date, using the form YYYY-MM-DD)
enddate (latest measurement date, using the form YYYY-MM-DD)
startarcdate (earliest archived date, using the form YYYY-MM-DD)
endarcdate (latest archived date, using the form YYYY-MM-DD

 

Depending on your desired output format, issue GET requests to one of the following (csv-text, HTML, or JSON, respectively):

 

https://seabass.gsfc.nasa.gov/search/sst.txt?
https://seabass.gsfc.nasa.gov/search/sst.html?
https://seabass.gsfc.nasa.gov/search/sst.json?

 

Examples:

 

https://seabass.gsfc.nasa.gov/search/sst.html?sensor=MODIS-Terra&sensor=MODIS-Aqua&startdate=2016-01-01

 

https://seabass.gsfc.nasa.gov/search/sst.json?sensor=MODIS-Terra&startdate=2016-01-01&enddate=2016-12-31

 

 

How to Download Files

Downloading files requires two steps. Your search query has returned a list of URLs of all the daily data files per sensor that met your criteria. You must parse through it (e.g., save the list locally and then loop through it) to download multiple data files.
 
1) If you used the web-browser search method, you may click individual files one at a time. However, to download all the results more efficiently, click the “Download as List” button to locally save a text file containing all the URLs. API-search users will similarly be working with a list.
 
2) To download files in bulk, use software of your choice to loop through the list of URLs and download them. NASA does not endorse particular software, but example solutions include using wget or curl from the command line, or else writing a script in a language like Python.
 
wget -i sst_results.csv --nocheck-certificate
get_files.py
 
Important note: Whatever method you use, files must be downloaded sequentially. Do not open many simultaneous connections or your connection will be blocked temporarily, or banned.
 
 

Data Format

SST match-up results are formatted as ASCII-text files using the SeaBASS File Format (slightly modified.) The file format consists of a block of metadata headers containing standardized keyword-value pairs and free-form comments (rows beginning with / and !, respectively.) The data are provided below the metadata headers arranged in a comma-delimited data matrix (i.e., similar to a spreadsheet.) Note that downloaded files are initially compressed, and must first be extracted.
 
 

File names always follow the pattern:

 

sstval_<YYYYMMDD>_<DOY>_<sensor>_<platform>_<#>pixl.tgz

 

<YYYYMMDD> is the measurement date

<DOY> is the measurement date expressed as DOY

<sensor> and <platform> are the satellite sensor and platform

<#> is the length of the size of a box of satellite pixels centered on the in situ target (i.e., 5 refers to satellite statistics generated from a 5x5 pixel box)

 

For example, sstval_20190103_003_VIIRS_SNPP_5pixl.tgz

   
Metadata headers contain summary information about the data in the file and processing information. More information. General tips for interpreting this information include:

 

Metadata “/fields” (and corresponding “/units”) are especially important, as they describe the columns of data in the data matrix. Field names in SST validation files are based on standardized SeaBASS field names for in situ values and l2gen parameter names for satellite products, but have been modified with additional prefixes and suffixes. As a side note, always refer to field names to interpret a column of data, and do not assume columns are always in the same order in every file.

 

A defined “/missing” value (e.g., -999) is used to replace values in the data matrix where data were missing or undefined.

 

Additional descriptive information can be found stored as comments in the metadata headers. Lines in the header beginning with the “!” character include free-form information, such as descriptions of satellite processing versions, and definitions of special terms used in the file.

 

A few small exceptions to the normal SeaBASS file format rules were made to adapt the file format for storing these results. Notably, the "units" of insitu_date_time contains a space, i.e., yyyy-mm-dd hh:mm:ss (whitespace is not normally allowed.) Other changes include the addition or subtraction of a few specific metadata headers (e.g., +platform, +instrument; -measurement_depth, -cruise, -documents, -calibration_files, -data_type).

 


In situ fields

 

The in situ field names are based on SeaBASS’s standardized list of field names modified by adding “insitu_” as a prefix. However, these files are a slightly modified version of the SeaBASS file format, with several new or repurposed in situ-related fields including:

 

    Table of special in situ field names used for SST

field name description
cruisename brief indicator of original in situ data provider
env_id arbitrary number
sat_id arbitrary number
insitu_cmcsst Canadian Meteorological Center (CMC) SST product
insitu_filename a reference for tracking data provenance
insitu_oiavhrr Optimum Interpolation Advanced Very High Resolution Radiometer (AVHRR) SST product
insitu_quality  0 = no quality control, 1 = passed general, 2 = final highest quality,  9 = bad
insitu_sample RSMAS numerical in situ code describing data source (defined in metadata header comments), e.g., 0 = moored buoy, 1 = drifting buoy, 2 = ship, 3 = MAERI, 4 = filter radiometer, 5 = Argo profiler, 6 = coral reef watch buoy
insitu_SN In situ ID., e.g., buoy ID codes, ship alpha-numeric call signs, MAERI instrument names and number, etc.

 


Satellite (l2gen output) fields


The fields in the file corresponding to satellite data or other output from l2gen are structured <prefix>_<l2gen-name>_<suffix> where:

 

<prefix> = <sensor>_<platform> such as MODIS_Aqua, MODIS_Terra, or VIIRS_SNPP
 
<l2gen-name> = l2gen product name
 
<suffix> = an additional description of the values or statistic contained in a data column. Satellite data are nominally derived from values of a 5x5 box of satellite pixels, and suffixes include: “center_pixel_value”, “median”, “stdev”, “min”, or “max”.

 

For example, MODIS_Aqua_sst_center_pixel_value or MODIS_Terra_bt_8550_median

 

 

Software tools

Several software tools are available for working with the SeaBASS file format, including Read Tools for opening data files in Python and MATLAB.
 
 

Background Information

 

Assembly of In Situ Data

Datasets of in situ SST measurements are compiled and analyzed at the University of Miami Rosenstiel School of Marine Science (RSMAS). Measurements come from a variety of sources including moored buoys, drifting buoys, ship cruises, MAERI, filter radiometers, argo profilers, and coral reef watch buoys. Information about the source of individual data records can be determined from the validation file under columns including “insitu_sample” (the numerical RSMAS in situ code), and “insitu_sn” (an in situ ID that corresponds to, for example, a particular buoy or ship).

 

 

Assembly of Satellite Data

L2gen software is used to process ocean color satellite data files to produce SST and other data products. Validation results are only created where satellite measurements were determined to be close enough temporally and spatially to be considered coincident with the in situ measurements. Satellite SST measurements are considered temporally coincident if satellite observations occurred within plus or minus approximately 30 minutes of in situ measurements. Spatial coincidence is evaluated with the following logic: satellite measurements are centered directly on the pixel encompassing the in situ measurement if the “qual_sst” value of that center pixel is zero (i.e., the qual_sst values range from 0-4, where 0 is best). However, if that pixel is not of the highest quality, then the satellite measurement is instead re-centered on the highest quality pixel that can be found closest within a 10 km radius of the in situ measurement (Minnet, 1991). Latitude and longitude coordinates for both in situ and satellite data center pixel values are provided in the results so that users may calculate the offset distance in case they wish to further filter the results.
 
If satellite data are found that meet these definitions of coincidence with the in situ data record, they are paired and provided on the same row in the data matrix of the validation results file. Only such successful match-ups are provided in the file. In situ points are excluded if no coincident satellite observations were found, or in cases where satellite pixels were so close to the edge of a granule that their 5x5 pixel match-up box was truncated and thus not considered a valid match-up. 
 
Notes on use: Prior to the in situ data being paired with satellite data, the in situ data have nominally been filtered to prevent oversampling situations where multiple high frequency measurements in space or time are paired with the exact same satellite data. However, some additional screening may be required. Users may also wish to further filter results based on qual_sst values and other factors, such as day/night flag, data source, insitu_quality, etc.
 
 
Minnett, P. J. (1991), Consequences of sea surface temperature variability on the validation and applications of satellite measurements, J. Geophys. Res., 96(C10), 18475–18489, doi:10.1029/91JC01816.
 
Tags:
Last edited by Chris Proctor on 2019-07-10
Created by Chris Proctor on 2019-03-27