GMTCO errors with SDR and VCM anc granulation

[jhuangadl]

Any hints on the following errors are greatly appreciated.
I checked the GMTCO files are not corrupted and they were converted to internal format successfully. What else could go wrong? Why file size is not correct?

Code: Select all

Running SDR ancillary granulation subroutines for Granule ID NPP000310953948
ProCmnDataItem[13ProCmnSDRItem]::setDataVectorExisting(VIIRS-MOD-GEO-TC) [0x4051de0] ROOT PRO_FAIL wrong file size: mapped file size 81103832 != expected size 81103784 from file ProCmnDataItem.cpp, line 2719
ProCmnAlgorithm[ProAncViirsGranulateTerrainGeopotentialHeight]::handleInputQuery(NPP000310953948) [0x3fd62e0] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-MOD-RGEO-TC , Granule ID: NPP000310953948 Failure(s):  (VIIRS-MOD-RGEO-TC, NPP000310953948), (VIIRS-MOD-GEO-TC, NPP000310953948) from file ProCmnAlgorithm.cpp, line 7880

Running VCM ancillary granulation subroutines for Granule ID NPP000310953948
ProCmnDataItem[13ProCmnSDRItem]::setDataVectorExisting(VIIRS-MOD-GEO-TC) [0x4af4ac0] ROOT PRO_FAIL wrong file size: mapped file size 81103832 != expected size 81103784 from file ProCmnDataItem.cpp, line 2719
ProCmnAlgorithm[ProAncViirsGranulateOpticalDepth]::handleInputQuery(NPP000310953948) [0x4af1830] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-MOD-RGEO-TC , Granule ID: NPP000310953948 Failure(s):  (VIIRS-MOD-RGEO-TC, NPP000310953948), (VIIRS-MOD-GEO-TC, NPP000310953948) from file ProCmnAlgorithm.cpp, line 7880

[tjensen]

I'm guessing what's going on here is that you are trying to run operational data that was produced using the Mx6.3 baseline using ADL 4.0, which does not include the Mx6.3 changes. There was a change made to the VIIRS RDRs which affected the VIIRS GEO products in Mx6.3 that resulted in a change in the product size, which causes the mismatch you're seeing.

Unfortunately, this means you can't run VIIRS GEO data produced using Mx6.3 in ADL 4.0. I believe you can run the VIIRS SDR algorithm using the Mx6.3 VIIRS RDRs and produce GEO products of the right size, but I can't vouch for the quailty of those GEO products (I'm not sure what the exact changes were and those that would know are out for the day).

ADL 4.1 includes the Mx6.3 changes and should be able to run using the newly sized GEO products. ADL 4.1 was shipped out of Raytheon on September 28 to Paul Meade, University of Wisconsin, and NASA. It looks like the DVDs are now available on the UW Download site, but no Virtual appliance as of yet. Last I heard they hoped to have it available sometime this week.

[jhuangadl]

Thank you, Tim,
At least I am glad to know why the error was caused.
So, this pretty much means ADL4.0 is having less value after Mx6.3 baseline, which pushes us to ADL4.1.

[lucy liu]

I gave a try using the data downloaded from CLASS on Aug. 26th, which should be generated under MX6.3.
And I am running in ADL4.1. I still got the similar error as shown below.

ProEdrViirsLstController processing did not complete successfully
ProCmnDataItem[13ProCmnSDRItem]::setDataVectorExisting(VIIRS-MOD-GEO-TC) [0x2aaab804fc40] ROOT PRO_FAIL wrongd size 81103832 from file ProCmnDataItem.cpp, line 2721
ProCmnAlgorithm[ProEdrViirsLst]::handleInputQuery(NPP000266808367) [0x2aaab804a6d0] ROOT PRO_FAIL Required inTC , Granule ID: NPP000266808367 Failure(s): (VIIRS-MOD-RGEO-TC, NPP000266808367), (VIIRS-MOD-GEO-TC, NPP000895

Please help, thanks.

[geoffc]

I just tried running ADL4.1 (via CSPP) on some GMTCO granule blobs that were generated using the ADL4.0 VIIRS SDR package and
unpacked using the ADL4.1 unpacker, and I got

Code: Select all

###########################
  Geolocation Information
###########################
N_Granule_ID :  NPP000331539514
ObservedStartTime :  2012-11-09 17:25:52.210029
URID :  50aa6662-e4cd4-6880946f-fe114595
N_Collection_Short_Name :  VIIRS-MOD-GEO-TC
###########################

geolocation asc file :  50aa6662-e4cd4-6880946f-fe114595.asc
geolocation blob file : 50aa6662-e4cd4-6880946f-fe114595.VIIRS-MOD-GEO-TC
We have long form geolocation names
Traceback (most recent call last):
  File "CSPP_VIIRS_EDR/CSPP/viirs/edr/adl_viirs_edr_masks.py", line 3749, in <module>
    sys.exit(main())
  File "CSPP_VIIRS_EDR/CSPP/viirs/edr/adl_viirs_edr_masks.py", line 3669, in main
    _getGRC(work_dir,anc_granules_to_process)
  File "CSPP_VIIRS_EDR/CSPP/viirs/edr/adl_viirs_edr_masks.py", line 1556, in _getGRC
    latitudeList,longitudeList,latMinList,latMaxList,lonMinList,lonMaxList,latCrnList,lonCrnList = _get_geo_Arrays(geoDicts)
  File "CSPP_VIIRS_EDR/CSPP/viirs/edr/adl_viirs_edr_masks.py", line 699, in _get_geo_Arrays
    geoBlobObj = adl_blob.map(geoXmlFile,geoBlobFileName,endian=endian)
  File "CSPP_VIIRS_EDR/CSPP/common/adl_blob.py", line 291, in map
    return _map(fp, struct, writable)
  File "CSPP_VIIRS_EDR/CSPP/common/adl_blob.py", line 210, in _map
    data = struct.from_buffer(mm)
ValueError: Buffer size too small (81103784 instead of at least 81103832 bytes

This looks related to jhuangadl's error.

[lucy liu]

I just tried using the data on Nov.5, 2012. And it runs ok. The previous data on Aug. 26 is built with Mx6.2 not Mx6.3, right?
So ADL4.1 can not process data before MX6.3, which causes some trouble in our validation work.

And I found that the output file name is changed due to the combination of geolocation information.
I am wondering where I can change this setting by only include the LST data? Thanks

[bhenders]

Lucy,

ADL 4.1 is sync'ed with Mx6.3 so it will run with all Mx6.3 operational data. I was told yesterday that the Geolocation format change went into Mx6.2, but have now learned it actually went into Mx6.3. Therefore, because of the mismatch in geolocation products ADL 4.1 is compatible with Mx6.3, ADL 4.0 is compatible prior to Mx6.3 geolocation data. I currently have a developer that is writing a conversion tool (he said it would take an hour to write), that will convert the gelocation products from one format to another. It will either insert the missing 48 bytes to the geolocation product binary file or remove it depending on the direction. These 48 bytes aren't actually used by any downstream algorithm so having them or not will not create any real side effects.

For your second question, I believe you are indicating the HDF5 file name now containing both the LST EDR and the geolocation product in the one file. This is controlled by the $ADL_HOME/cfg/ADL_CFG.xml file. The default setting was that HDF5 packaging is turned on.

To disable packaging change this entry in that file to be:

<config>
<!-- Option to enable HDF packaging. If enabled, the HDF output files are
created according to the data delivery packaging rules - multiple
products may be added to the same HDF file. An example is GEO
products, they will not be written to separate HDF files; instead,
they will be added to the SDR/EDR/IP HDF files.

There are 2 valid values:
true - Enable HDF packaging
false - Disable HDF packaging
-->
<name>ENABLE_HDF_PACKAGING</name>
<configValue>false</configValue>
</config>

Bryan Henderson
Raytheon Company

[lucy liu]

It is great to know you will provide a tool to convert the geolocation data built from earlier version.
So the tool will be available at ADL website, right?

Thanks very much for your response. Happy Thanksgiving.

[ppurcell]

FYI regarding what changed within the GEO file:

An additional quality flag was added sized 1 byte with an array length of 48 (=48 bytes total) to indicate the scan encoder electronics (SCE) side used.

See:
PCR031610 – PRO
474-CCR-12-0489
DR 4759

[kbisanz]

It is great to know you will provide a tool to convert the geolocation data built from earlier version.
So the tool will be available at ADL website, right?

Thanks very much for your response. Happy Thanksgiving.

I believe the conversion tool you're looking for is posted at viewtopic.php?f=27&t=283&sid=9365bc6c08 ... 8333#p1024