Outpath of ADL_Unpacker

[jhuangadl]

I am experimenting ADL4.0, and encounter this difference:
The ADL_Unpacker in ADL3.1 used to unpack NCEP/NAAPS h5 files to the current folder rather than to the HDF5_Unpack_Area default folder, although it unpacks those SDR h5 files to the HDF5_Unpack_Area folder, but
in the current ADL4.0 packge, the ADL_Unpacker unpack everything to the HDF5_Unpack_Area folder without discriminating NCEP/NAAPS or other h5 input files.
This starts messing up the DynAncConverter and PWconverter that need ADL_Unpacker to unpack the NCEP/NAAPS h5 files to the current folder then read the unpacked data to generate the internal formatted NCEP/NAAPS ancillary files.
How can I change the output path settings of the ADL_Unpacker to make it back to its operational mode as in its ADL3.1 environment? Thanks!

[kbisanz]

The short answer is that in ADL 4.0, everything that is unpacked now goes to the same area. You can change the area by changing "DMS_PATH" in /vobs/IDPS-CAT3/ADL/cfg/DDSADL_CFG.xml. However, everything will get unpacked to that new directory. NCEP/NAAPs products are no longer discriminated against.

If you're experimenting with ADL 4.0, I'd encourage you to check out section "5.9.8 Bringing New Externally Formatted Data into ADL" of $ADL_HOME/doc/UG60917-IDP-034A_IDPS_ADL_SW_Users_Manual_Part_2.pdf

In short, the DynAncConverter and PWconverter tools were really stopgap measures until full Ingest MSD (MSD = Mission Support Data) capability could be added. In ADL 4.0 Ingest MSD is available and should be the preferred way to ingest files and convert them from their external format to their internal format (with .asc files for metadata). It should convert dynamic ancillary data (NCEP/NAAPS), polar wander files, TLE files, processing coefficients, LUTs, etc. Basically everything the operational MSD process handles. Note that Ingest SMD (which process packets into RDRs) is a separate thing and not included in ADL.

Section 5.9.8 should hopefully answer a lot of questions, but please post back if something is not clear.

[pe_meade_dpe]

Although the DynAncConverter and PWConverter Tools were intended for an ADL 3.1 environment, and are not really necessary with ADL 4.0 built-in support for Mission Support Data, I will look into updating them for compatibility with the ADL 4.0 package. This should minimize the impact on users who have already established processes that rely on those JPSS DPE-created tools.

I do not currently have an ETA on this work.

[jhuangadl]

I am looking into the sortMsdtoLz.pl and runMsd.pl in the $ADL_HOME/script. I wish to share the following with you all:
1. First, the ADL_MSD_LZ_PATH was not set to any default by the envSetup.ksh, so I put it as ${ADL_HOME}/data/MSD_LZ
2. I downloaded the off_NCEP*.h5 files from CLASS
3. I unpacked these h5 file to $ADL_HOME/data/HDF5_Unpack_Area, then I ran sortMsdtoLz.pl, and it puts all the unpacked data to MSD_LZ folder
4. Then I run the runMsd.pl, it generated the internal format files successfully, but it removed all my h5 files in the landing zone automatically as well.
I am still testing everything out. It is annoying though my ADL_Unpacker.exe cause segmentation fault in a more prepared VA session, which I cannot fix it. I had to test all these by starting everything from scratch, and keep fingers crossed that my Unpacker will not go wrong again.

[kbisanz]

You're correct in setting ADL_MSD_LZ_PATH to be ${ADL_HOME}/data/MSD_LZ. It is set correctly when the code is shipped from Raytheon. It probably got accidentally changed when the virtual machine was created. I know UW has to update the envSetup file to work with the virtual machine (to update COTS locations and such).

When running runMsd.pl the expected and intended behavior is to remove files from the landing zone. The reason is to prevent files from being processed multiple times if runMsd.pl was run multiple times.

If you run ./sortMsdIntoLz.pl -h you can see the difference between the -c, -m, and -s options.

/vobs/IDPS-CAT3/ADL/script > ./sortMsdIntoLz.pl -h

Usage: ./sortMsdIntoLz.pl [-h] [-c|-m|-s] [-f] [-v] [-p] FILE ...

Required Options:
-c|-m|-s (Specify exactly one of -c -m -s):
-c Copy source files to landing zone directory
-m Move source files to landing zone directory
-s Symbolically link source files to landing zone directory
FILE File(s) to process. Shell pattern expansion is supported
(e.g. ~/classData/*.bin)

Basically files in the landing zone should be treated as "temporary". You shouldn't store your only copy of the file in the landing zone. As you found, they're removed after processing. I typically do something like this:

Code: Select all

./sortMsdIntoLz.pl -s ~/myData/*

If the data is stored in ~/myData/, that command will place a symbolic link inside the appropriate directory (ANC, AUX, or TILED) of $ADL_MSD_LZ_PATH. When runMsd.pl is run, the symbolic links inside of $ADL_MSD_LZ_PATH will be removed, but the files they point to (in ~/myData/) will remain untouched.

Also, when running runMsd.pl on NCEP files, you probably also want to use the -a option to make sure you perform temporal interpolation also.

Regarding the segmentation fault, I replied to your other post (viewtopic.php?f=24&t=254). Unfortunately I had more questions than answers.

[jhuangadl]

Thanks a lot for the tips, Kevin,
It worked out beautifully, except I have to unpack the h5 files under another VA environment.
Cheers,
jingfeng

[Nima]

I am looking into the sortMsdtoLz.pl and runMsd.pl in the $ADL_HOME/script. I wish to share the following with you all:
1. First, the ADL_MSD_LZ_PATH was not set to any default by the envSetup.ksh, so I put it as ${ADL_HOME}/data/MSD_LZ

Hi jhuangadl
Where did you set up ${ADL_HOME}/data/MSD_LZ? in envsetup.ksh? or sortMsdtoLz.pl? I made the change in the latter and got this
"Error: $home/scientist/CSPP/ADL/data/MSD_LZ/ANC is not a writable directory"
Should I set this in envsetup.ksh and then do ./envsetup.ksh? And then source it? Any other steps?
Thanks for the info already posted

2. I downloaded the off_NCEP*.h5 files from CLASS
3. I unpacked these h5 file to $ADL_HOME/data/HDF5_Unpack_Area, then I ran sortMsdtoLz.pl, and it puts all the unpacked data to MSD_LZ folder
4. Then I run the runMsd.pl, it generated the internal format files successfully, but it removed all my h5 files in the landing zone automatically as well.
I am still testing everything out. It is annoying though my ADL_Unpacker.exe cause segmentation fault in a more prepared VA session, which I cannot fix it. I had to test all these by starting everything from scratch, and keep fingers crossed that my Unpacker will not go wrong again.

[tjensen]

Hi Nima,

I'd suggest adding this to your envSetup.ksh, if it's not there already:
export ADL_MSD_LZ_PATH=${ADL_HOME}/data/MSD_LZ

As Kevin said above, this is included in the file that's delivered from Raytheon, but appears to have been removed in the Virtual Appliance.

It looks like what you added may be setting it though, assuming your $ADL_HOME is set to "$home/scientist/CSPP/ADL/". Double check that $home/scientist/CSPP/ADL/data/MSD_LZ/ANC a directory you have write permissions to. If you don't have permissions and can't change the directory to add permissions, you may need to create a new Landing Zone area and point your ADL_MSD_LZ_PATH variable to it. ADL_MSD_LZ_PATH just needs to point to a directory with the following subdirectories:

> ls -1 $ADL_MSD_LZ_PATH
ANC
AUX
TILED
Temp

These subdirectories are where your unpacked h5 data will get put by the sortMsdIntoLz.pl and where the runMsd.pl script will look for inputs to convert.