how to input directories into ADL5.1

[wzchen]

I try to run the ADL5.1. Do I have to use database in ADL5.1? or do I have an option to use the old fashion way?

I used to keep all input directories in the "INFTK_DM_ROOT" variable and simply separated them by ":". However, in the new ADL, it seems that I can't do this anymore. Is there any simple way to input all input directories into database?

[16:03 weizhong@rhw3022 script]$ $ADL_HOME/script/DMS/createSQLiteDMS.csh
This script creates the SQLite DMS for ADL...
INFTK_DM_ROOT=/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-TLE/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-PolarWander/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-21:/data/data020/weizhong/CSPP_CM/ADL51_BLK2_snap/home/pub/ClearCase/STAR/JPSS/ADL_BLK2/LUTs:/data/data020/pub/NPP_DATA/LUTs/AllLuts_BLK2_ADL5.0_Beta/ANC:/data/data020/pub/NPP_DATA/TILES/NovGroundCCR692Tiles/with-meta-all::::/data/data048/pub/weizhong/temp/run_test_viirs/data/input/withMetadata/ProControllerInputs_VIIRS-SCIENCE-RDR/2012-01-21/d20120121_t0722063_e0723316_b00000/run_set:/data/data048/pub/weizhong/temp/run_test_viirs/data/output/withMetadata/ProSdrViirsCalControllerOutputs/20120121-MDR28_J1_VSDR_test/d20120121_t0722063_e0723316_b00000

/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-TLE/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-PolarWander/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-21:/data/data020/weizhong/CSPP_CM/ADL51_BLK2_snap/home/pub/ClearCase/STAR/JPSS/ADL_BLK2/LUTs:/data/data020/pub/NPP_DATA/LUTs/AllLuts_BLK2_ADL5.0_Beta/ANC:/data/data020/pub/NPP_DATA/TILES/NovGroundCCR692Tiles/with-meta-all::::/data/data048/pub/weizhong/temp/run_test_viirs/data/input/withMetadata/ProControllerInputs_VIIRS-SCIENCE-RDR/2012-01-21/d20120121_t0722063_e0723316_b00000/run_set:/data/data048/pub/weizhong/temp/run_test_viirs/data/output/withMetadata/ProSdrViirsCalControllerOutputs/20120121-MDR28_J1_VSDR_test/d20120121_t0722063_e0723316_b00000: No such file or directory.

Thanks,

[kbisanz]

You have to use the database in ADL 5.1 and newer.

To use the new way, set INFTK_DM_ROOT to a directory where you want to store the binary data. Then run $ADL_HOME/script/DMS/createSQLiteDMS.csh like you did. That script will create a new SQLite database at $DB_NAME (to store the metadata) and will create a directory structure under $INFTK_DM_ROOT to store the binary data. It might take a couple of minutes to run.

The easiest way to transfer data from a pre-5.1 DMS (a set of directories) to a 5.1 DMS (database + storage area) is to use $ADL_HOME/script/importExistingFlatFiles.csh. You need to provide that script the list of directories from which data should be in imported. The directories should be separated by spaces on the command line, not colons. Under the hood, that script will combine the .asc and binary files into saveset tar files and then import them into DMS using ImportIntoDms.plx.

When you run an algorithm, the binary output will be saved to DMS at the location printed in the status messages at the end of algorithm execution (if you're running it directly; the chain runners mask that output). Any HDF outputs will be written to $HDF_OUTPUT_PATH/AlgName/.

Since the binary outputs aren't stored in a single directory in ADL 5.1, the best way to determine what has been produced "lately" is to perform a query based on time. For example, you can find items with a CreateDateTime (CDT) greater than (GT) a given time by using the -gtcdt flag to
RetrieveFromDms.plx

Code: Select all

$ADL_HOME/script/DMS/RetrieveFromDms.plx -gtcdt "2015-10-13 20:47:45.000000Z"

the current time can be printed with:

Code: Select all

date -u +"%F %H:%M:%S.000000Z"

Note, in ADL 5.3 (not released yet) importExistingFlatFiles.csh was removed. This is because in ADL 5.3 the DMS import gained the ability to import from .asc and binary files directly (not just from .tar files).

[wzchen]

Is there any similar script to import multiple directories after ADL5.3?
Thanks,

[kbisanz]

In ADL 5.3 you can give ImportIntoDms.plx a directory which contains .tar or .asc/bin files and it will import into DMS. In ADL 5.1 and 5.2, it could only import .tar files. The reason why importExistingFlatFiles.csh was created in ADL 5.1 was to package the .asc/bin files into tar files so they can be imported by ImportIntoDms.plx. Since ImportIntoDms.plx can import .asc/bin files directly in ADL 5.3, importExistingFlatFiles.csh is no longer needed.

To import from multiple directories in ADL 5.3, you can use a script or shell loop and call ImportIntoDms.plx. In the bash shell it would be something like this:

Code: Select all

~ > for dir in dir1 dir2 dir3
> do                         
> $ADL_HOME/script/DMS/ImportIntoDms.plx -path $dir
> done 

In csh and tcsh shells it would be:

Code: Select all

~ > foreach dir ( dir1 dir2 dir3)                                    
foreach? $ADL_HOME/script/DMS/ImportIntoDms.plx -path $dir                      
foreach? end

importExistingFlatFiles.csh was deleted in ADL 5.3 because it could be replaced by a simple shell loop in an effort to reduce unneeded scripts and reduce ADL testing effort when we ship out a delivery.

[wzchen]

I got the following error when I ran createSQLiteDMS.csh

/data/data020/weizhong/CSPP_CM/ADL51_BLK2_snap/home/pub/ClearCase/STAR/JPSS/ADL_BLK2/ADL/tools/bin/DmHmiInitialize.exe: error while loading shared libraries: libDmCoreDb.so: cannot open shared object file: No such file or directory

Do you have any idea where this file is?
Thanks,

[kbisanz]

That library should be in $ADL_HOME/lib and should have been created when doing a "make library" or running a build script.

If the library does not exist try this:

Code: Select all

cd $ADL_HOME/CMN/Utilities/DMS/Core/Db/src
make library

If you have problems building in $ADL_HOME/CMN/Utilities/DMS/Core/Db/src my guess would be your SQLite COTS is not set up correctly. Paul Meade noticed a possible minor issue with that in ADL 5.1, so let me know if you have issues.

If the library does exist, check your $LD_LIBRARY_PATH and make sure you've source'd the envSetup file.

[wzchen]

Could you please provide me a list of all available "N_Dataset_Type_Tag" ? Thanks,

[kbisanz]

The available N_Dataset_Type_Tag values are: ANC, ANC-Gran, ANC-Int, AUX, AUX-Int, AUX-Ext, DQN, EDR, GEO, GridIP-Gran, GridIP-Gran-Int, GridIP-Tile, GridIP-Tile-Int, IP, RDR, SDR, TDR, or Verified-RDR. Most of them are fairly obvious. LUTs, processing coefficients, and DQTTs are all type AUX.

I updated the usage of $ADL_HOME/script/DMS/InsertIntoDms.plx to reflect the available types in the ADL 5.3 version that will be delivered.