how to use ADL_Packer?

[ljiang]

This is Lide Jiang from STAR (VIIRS/EDR/OCC). I am trying to understand how to use ADL_Packer to convert the output binary files into HDF5 formats. I successfully build the ADL3.0 beta and test-ran it. I got binary outputs which I want to convert to HDF5 format. I then followed the usage by typing:

ADL_Packer.exe 4e1c882b-b73b3-ca0aa1e4-f2a132f2 ../data/output/withMetadata/occOutputs/ ../cfg/ProEdrViirsOCC_GuideList.cfg

It shows "Finished." but I cannot find any hdf5 outputs anywhere. under ../data/output/withMetadata/occOutputs/ I have:

-rwxrwxrwx 1 lidej orbit 10645 Jul 12 13:48 4e1c882b-b73b3-ca0aa1e4-f2a132f2.asc
-rwxrwxrwx 1 lidej orbit 174489600 Jul 12 13:48 4e1c882b-b73b3-ca0aa1e4-f2a132f2.VIIRS-OCC-EDR
-rwxrwxrwx 1 lidej orbit 7831 Jul 12 13:48 4e1c88e3-3dbe0-ca0aa1e4-f2999437.asc
-rwxrwxrwx 1 lidej orbit 174489600 Jul 12 13:48 4e1c88e3-3dbe0-ca0aa1e4-f2999437.VIIRS-OCC-EDR.BE

What am I missing here?

[bhurley]

Hi,

If an algorithm is run using the "withMetadata" version of its execution configuration file (located in $ADL_HOME/cfg/dynamic/withMetadata/), the algorithm will produce both the internal binary versions and HDF versions of its deliverable output products.

An example would be: $ADL_HOME/bin/ProEdrViirsOCC.exe ../cfg/dynamic/withMetadata/ProEdrViirsOCCLwFile.xml

If you don't see HDF outputs, then there is likely a problem somewhere.

To correctly run the ADL_Packer, the environment variable "INFTK_DM_ROOT" must be set to the directory containing the data product and metadata file-pair you wish to pack. In your case, it looks like you want to set it to "../data/output/withMetadata/occOutputs" ... and it would probably be best to use the full path instead of a relative path. Additionally, the last command-line argument (config guide) should be the filepath to the DDS_DDSUTIL_GuideList.cfg, not algorithm's config guide. So you will want to use "$ADL_HOME/cfg/DDS_DDSUTIL_GuideList.cfg".

I have pasted a section from the draft User Manual describing the HDF Packer tool below. The documentation was not complete enough to include with the beta release, but it will of course be included with the official release. The info below should be helpful.

5.8.11 HDF Packer
The HDF Packer utility is a stand-alone command-line program for packing (or combining) an IDPS internal data product binary file and its associated metadata ASCII file (both having been produced in an IDPS or ADL system) into an HDF5 file format that conforms to a defined JPSS deliverable data product format. These formats are defined in the CDFCB-X (see Section 2 Reference Documents, Table 2-1, for CDFCB-X document number and current revision).

The HDF Packer will only create HDF5 files from IDPS internal data product and metadata file-pairs that are classified as JPSS deliverable data products. For example, most SDR and EDR products are deliverable, but FSDR and FEDR products (unscaled, full floating-point products) are not deliverable. For a complete list deliverable products, consult the CDFCB-X. The HDF Packer will also pack the shell versions of deliverable products. In ADL a shell product only consists of metadata, so the resulting HDF5 file will contain metadata only. The environment variable $INFTK_DM_ROOT must be set to the directory containing the data product and metadata file-pair you wish to pack. The output HDF5 file will be named according to the naming convention defined in the CDFCB-X for JPSS deliverable data products.

Program usage is described below:
ADL_Packer.exe <product URID> <output directory> <DDS config guide list>

Required Arguments:
- <product URID>: The URID of the product to be packed
- <output directory>: The directory (including the trailing “/”) where the HDF5 file will be written
- <DDS config guide list>: The filename, including the full path, of the DDS Configuration Guide List file. This file contains the list of Configuration Guides used by the utility.

An example of using the HDF Packer utility is illustrated below:
The environment variable $INFTK_DM_ROOT has been set to the directory containing the input data (data product and metadata file-pair). In this example $INFTK_DM_ROOT is set to “/home/adl_user/Packer_Input” and contains the following files:
- 4decf928-c6a31-eaed9b9d-3875c1e6.VIIRS-IST-EDR (binary data product file)
- 4decf928-c6a31-eaed9b9d-3875c1e6.asc (ASCII metadata file)

The directory “/home/adl_user/Packer_Output/” is the directory we want the output HDF file stored; this will be specified as a command-line argument.

Type the following on the command-line:
% $ADL_HOME/bin/ADL_Packer.exe 4decf928-c6a31-eaed9b9d-3875c1e6 /home/adl_user/Packer_Output/ $ADL_HOME/cfg/DDS_DDSUTIL_GuideList.cfg

The following messages will be sent to the terminal:
Finished

The output directory “/home/adl_user/Packer_Output/” now contains the following file:
- VISTO_npp_d20030125_t0722288_e0723533_b00014_-_devl_dev.h5

The HDF5 files will be named according to the naming convention defined in the CDFCB-X for JPSS deliverable data products. This file naming convention is described in the CDFCB-X, Vol 1, Section 3.4.1 and Section 3.4.4. A log file will also be written to the directory “$ADL_HOME/log/” for each invocation of the executable. The log file will be named “Packer_*”, where “*” is a series of alphanumeric characters.

All products that can be processed by the HDF Packer are effectively listed in the file “$ADL_HOME/CMN/Utilities/DDS/cfg/DDS_PRODUCT_IDS.xml”.

[houchin]

Required Arguments:
- <product URID>: The URID of the product to be packed
- <output directory>: The directory (including the trailing “/”) where the HDF5 file will be written
- <DDS config guide list>: The filename, including the full path, of the DDS Configuration Guide List file. This file contains the list of Configuration Guides used by the utility.

No other unix app that I can think of requires a trailing slash on directories specified on a command line. I can only see this causing problems. Please treat this as a bug to be corrected in ADL 3.next.

[ljiang]

thanks, bhurley for your detailed explanation. Another question: does it make any difference if I use the little-endian binary or big-endian binary file as the input to ADL_Packer?

[bhurley]

Binary inputs to the ADL_Packer should be of the same endian-type as the runtime platform. I believe this is what the ADL_Packer assumes. For example, if you are running on a x86 (little-endian) Linux platform, you should use the little-endian binaries as input to the ADL_Packer. This clarification will be added to the documentation. Thanks.

[bhurley]

Required Arguments:
- <product URID>: The URID of the product to be packed
- <output directory>: The directory (including the trailing “/”) where the HDF5 file will be written
- <DDS config guide list>: The filename, including the full path, of the DDS Configuration Guide List file. This file contains the list of Configuration Guides used by the utility.

No other unix app that I can think of requires a trailing slash on directories specified on a command line. I can only see this causing problems. Please treat this as a bug to be corrected in ADL 3.next.

Agreed. This will be added to a list of improvements for the next release.

[ljiang]

Binary inputs to the ADL_Packer should be of the same endian-type as the runtime platform. I believe this is what the ADL_Packer assumes. For example, if you are running on a x86 (little-endian) Linux platform, you should use the little-endian binaries as input to the ADL_Packer. This clarification will be added to the documentation. Thanks.

thanks for clarification, originally Ithought ADL_Packer can adjust this according to 'ByteOrder' info in metadata. So it's not that clever.