Satellite Validation Match-Up Tools (Portable)

These validation match-up tools allows Users to
  1. find relevant OB.DAAC satellite granules from in situ points or a SeaBASS data file and
  2. generate coincident satellite validation match-ups for those points.
These tools are downloadable for Users to run individually, external to the official SeaBASS validation search. These tools are designed to replicate some of SeaBASS validation search's core functionality, with the exception that these tools do NOT adjust in situ data to water-leaving values.
 
These validation match-up tools are served as part of SeaDAS and apply the default recommended match-up exclusion criteria, as set forth in S.W. Bailey and P.J. Werdell, "A multi-sensor approach for the on-orbit validation of ocean color satellite data products", Rem. Sens. Environ. 102, 12-23 (2006).



Getting Started

The satellite validation match-up tools are command-line tools served as part of the SeaDAS software package, which can be downloaded and installed from here. The SeaDAS software package does NOT buiild these tools by default, but they can be added via a few extra steps described below.

Install SeaDAS

Follow the directions for your system architecture listed here.

Additional installation steps

With SeaDAS installed and running:

  1. From the "OCSSW" dropdown menu, select "Install OC Processors."
  2. Under the "Others" section, check the "Source Code" box.
  3. Click "Run" which will create the "ocssw" source code directory within your SeaDAS installation directory.
  4. Add $OCSSWROOT to your shell environment.
    1. If you use BASH, execute these lines of code OR add them to .bashrc or .profile:
      export OCSSWROOT="[path_to_seadas_installation_folder]/ocssw"
      source $OCSSWROOT/OCSSW_bash.env
    2. If you are using a shell other than BASH, talk to your system admin.

Software components

The specific scripts to use for creating satellite validation match-ups are:
NOTE: The mk_matchup.py script uses this binary:

Satellite Data Finder

This script performs API-searches of the EarthData Common Metadata Repository (CMR) for coincident satellite granule names and download links, given
  1. an OB.DAAC satellite/instrument name and
  2. a latitude, longitude,  and time point or range,
    or a SeaBASS file with latitude, longitude, and date-time information as field.

Output

  1. a list of OB.DAAC L2 satellite file granule names that contain the input criteria, per the CMR's records.
  2. a list of granule download links to fetch the matching satellite file granules, per the CMR's records.

Input arguments

  -h, --help            show this help message and exit
  --sat {modisa,modist,viirsn,goci,meris,czcs,octs,seawifs}
                              String specifier for satellite platform/instrument
                              
                              Valid options are:
                              -----------------
                              modisa  = MODIS on AQUA
                              modist  = MODIS on TERRA
                              viirsn  = VIIRS on NPP
                              meris   = MERIS on ENVISAT
                              goci    = GOCI on COMS
                              czcs    = CZCS on Nimbus-7
                              seawifs = SeaWiFS on OrbView-2
                              octs    = OCTS on ADEOS-I
                              
  --data_type {oc,iop,sst}
                              OPTIONAL: String specifier for satellite data type
                              Default behavior returns all product suites
                              
                              Valid options are:
                              -----------------
                              oc   = Returns OC (ocean color) product suite
                              iop  = Returns IOP (inherent optical properties) product suite
                              sst  = Returns SST product suite (including SST4 where applicable)
                              
  --slat SLAT                 Starting latitude, south-most boundary
                              If used with --seabass_file, will override lats in the file
                              Valid values: (-90,90N)
                              
  --elat ELAT                 Ending latitude, north-most boundary
                              If used with --seabass_file and --slat, will override lats in the file
                              Valid values: (-90,90N)
                              
  --slon SLON                 Starting longitude, west-most boundary
                              If used with --seabass_file, will override lons in the file
                              Valid values: (-180,180E)
                              
  --elon ELON                 Ending longitude, east-most boundary
                              If used with --seabass_file and --slon, will override lons in the file
                              Valid values: (-180,180E)
                              
  --stime STIME               Time (point) of interest in UTC
                              Default behavior: returns matches within +/- MAX_TIME_DIFF (default +/-3 hours) about this given time
                              If used with ETIME, this creates a search time window, between STIME and ETIME.
                              Valid format: string of the form: yyyy-mm-ddThh:mm:ssZ
                              OPTIONALLY: Use with --max_time_diff or --etime
                              
  --max_time_diff MAX_TIME_DIFF
                              Maximum time difference between satellite and in situ point
                              OPTIONAL: default value +/-3 hours
                              Valid values: decimal number of hours (0-36)
                              Use with --seabass_file OR --stime
                              
  --etime ETIME               Maximum time (range) of interest in UTC
                              Valid format: string of the form: yyyy-mm-ddThh:mm:ssZ
                              Use with --stime
                             
  --seabass_file SEABASS_FILE [SEABASS_FILE ...]
                              Valid SeaBASS file name or list of file names
                              File must contain latitude, longitude, and date-time information as fields.
                              
  --get_data GET_DATA         Flag to download all identified satellite granules.
                              Requires the use of an HTTP request.
                              Set to the desired output directory.
 

Usage examples

         fd_matchup.py --sat=modist --slat=23.0 --slon=170.0 --stime=2015-11-16T09:00:00Z --time_window=8

         fd_matchup.py --sat=modist --stime=2015-11-15T09:00:00Z --etime=2015-11-17T09:00:00Z --slat=23.0 --elat=25.0 --slon=170.0 --elon=175.0

         fd_matchup.py --sat=modist --time_window=4 --seabass_file=[your SB file name].sb

Satellite Match-Up Extractor

 This script creates satellite extracts for a given SeaBASS file, saving the values passing the specified exclusion criteria to a SeaBASS file. This script applies Level-2 satellite match-up methodology of Bailey and Werdell (2006), with a few exceptions:
  1. User-defined inupts to mk_matchup.py will override recommended defaults
  2. The maximum allowed solar zenith angle used here is 70-deg vs the Bailey & Werdell recommended 75-deg
  3. The Rrs and AOT data are only in the OC L2 satellite product suite, if using standard OBPG-distributed products. Other file_types (SST, SST4, IOP, etc) will not evaluate any maximum coefficient of variation threshold.
  4. The maximum coefficient of variation theshold is derived by taking the median, in spectral-space, of the band-specific CVs, derived by dividing the filtered standard deviation by the filtered mean of the Rrs and AOT from the, nominally, 5 by 5 extract. This is done for all Rrs bands between 405nm and 570nm (omitting the 469 and 555 land bands from MODIS) and for all AOT between 860nm and 900nm.
  5. For all SST L2 satellite product suite data, the qual_sst_max or qual_sst_mean fields should be used to screen the sst value quality. The qual_sst value varies between 0 (best) and 4 (worst), where qual_sst_mean (qual_sst_max) is the mean (max) of the 5 by 5 pixel satellite extract.

Output

  1. the original SeaBASS data
  2. coincident satellite products as additional columns into --out_file
    OR if --out_file is not specified, --seabass_file with "_matchups.sb" appended

REQUIRED input arguments

  1. --sat_file SAT_FILE [SAT_FILE ...]
                                  REQUIRED: input OB.DAAC Level-2 satellite netCDF file(s)
  2. --seabass_file SEABASS_FILE [SEABASS_FILE ...]
                                  REQUIRED: input SeaBASS file(s)
                                  Must be a valid SeaBASS file, passing FHCHECK with no errors.
                                  Matched-up satellite variables will be appended as additional fields to the data matrix and relevant headers.
                                  File must contain latitude and longitude and date-time expressed as FIELD entries.

OPTIONAL input arguments

  -h, --help            show this help message and exit
                              
  --box_size BOX_SIZE         OPTIONAL: box size of the satellite data extract made around the in situ point
                              Valid values are odd numbers between 3 and 11, default = 5
                              
  --min_valid_sat_pix MIN_VALID_SAT_PIX
                              OPTIONAL: percent minimum valid satellite pixels required to create an extract
                              Valid value: (0.0 - 100.0), default = 50.0
                              
  --max_time_diff MAX_TIME_DIFF
                              OPTIONAL: maximum time difference between satellite and in situ point
                              Valid value: decimal number of hours (0 - 36 hours), default = 3
                              
  --max_coeff_variation MAX_COEFF_VARIATION
                              OPTIONAL: maximum coefficient of variation of satellite pixels within the satellite extract
                              Valid value: (0.0 - 1.0), default = 0.15
                              
  --slat SLAT                 OPTIONAL: Starting latitude, south-most boundary
                              If used with --seabass_file, will override lats in the file
                              Valid values: (-90,90N)
                              
  --elat ELAT                 OPTIONAL: Ending latitude, north-most boundary
                              If used with --seabass_file and --slat, will override lats in the file
                              Valid values: (-90,90N)
                              
  --slon SLON                 OPTIONAL: Starting longitude, west-most boundary
                              If used with --seabass_file, will override lons in the file
                              Valid values: (-180,180E)
                              
  --elon ELON                 OPTIONAL: Ending longitude, east-most boundary
                              If used with --seabass_file and --slon, will override lons in the file
                              Valid values: (-180,180E)
                              
  --verbose                   OPTIONAL: Displays reason for failed matchup for each in situ target called.
                              
  --no_header_comment         OPTIONAL: Flag to NOT append exclusion criteria to the OFILE header. Useful when running script repeatedly.
 

Notes on OPTIONAL input arguments

  1. --slat AND --slon    must be used together and will override any latitudes and longitudes in --seabass_file
  2. --elat AND --elon   must be used together and with --slon and --slat
These will override any latitudes and longitudes in --seabass_file and use a latitude and longitude bounding box instead of --box_size

Usage examples

         mk_matchup.py --sat_file=[file name].nc --seabass_file=[file name].sb --out_file=[OPTIONAL, file name].sb
         mk_matchup.py --sat_file=[file name].nc --seabass_file=[file name].sb --slat=45.3 --slon=-157.4
         mk_matchup.py --sat_file=[file name].nc --seabass_file=[file name].sb --slat=45.3 --elat=48.7 --slon=-157.4 --elon=-145.3

References

Bailey, S.W., and Werdell, P.J. (2006). A multi-sensor approach for the on-orbit validation of ocean color satellite data products. Rem. Sens. Environ. 102, 12-23.

Last edited by David Norris on 2019-09-13
Created by David Norris on 2017-03-31