File names for binary LUT files in withMetadata mode

[houchin]

Hi again,

I am looking at the ADL sample inputs and have a question about the filenames for the binary LUT files when running in withMetadata mode.

For example, in the ProSdrViirsControllerInputs, the Delta C LUT file is named:

4d7ee095-929f5-0a4f180b-57d6dc05.VIIRS-SDR-DELTA-C-LUT_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_PS-1-D-NPP-3-PE-_devl_dev_all-_all.bin

Beyond the URID.CSN, is any of the rest of that required? Or would I be just fine if the file was just named

4d7ee095-929f5-0a4f180b-57d6dc05.VIIRS-SDR-DELTA-C-LUT

[kbisanz]

A name in the form of URID.CSN should work. I believe that the algorithm will function fine. The "AncFilename" metadata on the output will be different. Of course if you name a file URID.CSN, the file name in the URID.asc file should match also.

If/when ADL has utilities to actually ingest files, the file name will matter. However, it does not matter to the current ADL version.

[houchin]

One more question. Is there an "easy" way to unpack a binary LUT file to generate the .asc file for it, or in effect is this a fully manual process (copy an existing one and update it with any new parameters and the new URID)?

[JimBiard]

The binary blob files don't contain any metadata. So your copy and modify approach is close to being it. You can write an application using ADL in which you access a binary LUT file in withMetadata mode and write it out again, modifying the metadata on the output, but that may be a bit heavy for your purposes. I just noticed the UridCreator.exe application that is built in ADL 3.0. That will help a lot with creating a new URID.

[houchin]

I looked in one of the sample metadata files (for the VIIRS Delta C LUT), and it looks like all of the data in it is pretty generic. I'm going to head down the path of just building it from scratch when I have to generate a URID for a new input file.

[kbisanz]

If copying a .asc file from a provided .asc file, don't forget to add the "ByteOrder" metadata. ByteOrder is an ADL only piece of metadata which specifies if the file is big or little endian.

Because the sample input and truth output is from an IDPS DMS, (where ByteOrder is not a valid piece of metadata) ByteOrder does not exist in the sample input and truth output .asc files. If the code does not find ByteOrder, it assumes that it is big endian. Note that this assumption is different from running in no metadata mode where not finding "DataEndianType" in the algorithm's *_CFG.xml file assumes native endianness.

The ADL code will add ByteOrder to all .asc files produced in ADL. The line to add will be one of the following, depending on if the binary is big or little endian.
("ByteOrder" STRING EQ "BE")
("ByteOrder" STRING EQ "LE")
The location at which it's added doesn't matter, so long as it's within the "METADATALIST COLLECTION" block in the .asc file.

In conclusion, if you're making a new .asc file don't forget to add ByteOrder, or else the code will assume big endian, which is not what you want if the binary LUT is created on a little endian platform (because it would then be unnecessarily byte swapped).

*Edit: If you're creating a new LUT, it may be helpful to read section "5.9.1 Creating and Using a New Input or Output" of User Manual Part 2.

[houchin]

While the whole endian issue is unfortunate (couldn't everyone just have gotten along in the first place), I think that persisting files in multiple endians, especially when the filename convention contains a mode where the endian cannot be determined without knowing the origin, is a bad idea.

For the Aerospace tools I've created, we are only writing big endian LUTs. Yes, it is unfortunate that swapping is necessary, but I believe that robustness is more valuable than the slight performance improvement by writing little endian data. I would encourage others to adopt a similar convention.

This is not unheard of in other industries. For example, JPEG 2000, JPEG and MP4 files are all fixed at big endian.