ADL 4.0 Ingest NCEP grib data

[kbisanz]

There shouldn't be any hidden timestamps that are being checked (at least that come to mind ).

From a quick look at your log, it appears that this is your problem:

Code: Select all

2012/11/02 17:55:20.937.053 (26140.47933412367008): DBG_LOW DmCoreURMetadataFilter.cpp|149| tid-47933412367008 DMS query start: Searching for items matching the following metadata:
Metadata Collection METADATALIST (size 3)
item 0, 0x7e61f48 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Collection_Short_Name, comparison=EQ, value=NCEP-ANC-Int
item 1, 0x7e49a00 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=INTEGER name=DatasetLock, comparison=EQ,value=0
item 2, 0x7e55b70 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=DATETIMERANGE name=Effectivity, comparison=EQ2012-01-03 09:29:25.542000,2012-01-03 09:29:25.542000

2012/11/02 17:55:20.945.562 (26140.47933412367008): DBG_MED DmCoreURMetadataFilter.cpp|258| tid-47933412367008 DMS query finish: query returned 0 item(s) containing these items:

If the code finds an item matching those 3 pieces of metadata, it satisfies the query and an item would be returned.

Earlier you pasted this:

Code: Select all

5093f93b-38c91-229b0fa9-8c2d70d5.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 08:30:00.000000" "2012-01-03 09:29:59.999999")
5093fa8e-8e77e-229b0fa9-33cde765.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 09:45:00.000000" "2012-01-03 10:29:59.999999")
5093f93b-41ecb-229b0fa9-63b4069f.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 11:30:00.000000" "2012-01-03 12:29:59.999999")
5093fa8e-a8de3-229b0fa9-33cf8dca.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 10:30:00.000000" "2012-01-03 11:29:59.999999")

My first thought was that the Effectivity was not matching, however the point of "2012-01-03 09:29:25.542000" is within the range of "2012-01-03 08:30:00.000000" "2012-01-03 09:29:59.999999" in 5093f93b-38c91-229b0fa9-8c2d70d5.asc. Please verify that 5093f93b-38c91-229b0fa9-8c2d70d5.asc satisfies these two additional pieces of metadata:

Code: Select all

item 0, 0x7e61f48 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Collection_Short_Name, comparison=EQ, value=NCEP-ANC-Int
item 1, 0x7e49a00 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=INTEGER name=DatasetLock, comparison=EQ,value=0

The TIME_UNTIL_USE only gets checked when you perform temporal interpolation. It looks like you're using DynAncInterpolator.pl which is not a Raytheon script, so I can't guarantee what it does. That script was a stopgap measure to provide temporal interpolation capability until Raytheon could convert the operational temporal interpolation to ADL. I don't know exactly when Raytheon completed that, but I know it was in ADL 4.0.

In order for TIME_UNTIL_USE to make a difference (assuming DynAncInterpolator.pl checks it), you would have had to rerun DynAncInterpolator.pl and verify that Effectivity changed from "2012-01-03 09:45:00.000000" "2012-01-03 10:29:59.999999" to "2012-01-03 09:30:00.000000" "2012-01-03 10:29:59.999999". I apologize if I didn't make that clear before. But that doesn't seem to be your immediate problem anyway because 5093f93b-38c91-229b0fa9-8c2d70d5.asc would seem to have the desired effectivity range.

I also notice from looking at your log that there does not seem to be a file for the URID 5093f93b-38c91-229b0fa9-8c2d70d5. I can tell this from searching for "URs in inventory" and not finding "5093f93b-38c91-229b0fa9-8c2d70d5" in the list. I would verify that the directory containing 5093f93b-38c91-229b0fa9-8c2d70d5.* is in the input path set in the ProEdrCrimssAtmProfControllerLwFile.xml

[rnc]

OK, thanks for the explanation of the Effectivity of the 3HR NCEP. The problem turned out to be that one of the NCEP files had N_Collection_Short_Name set to NCEP-03HR-Anc-Int and thus failed the test you pointed out. I fixed it (for now) by changing the files used for interpolation to avoid the 03HR product. This does not match what's in the GCRIO-REDRO files from CLASS, but I've inferred the pattern there and will work on implementing something that matches better the operational algorithm.

The EDRs in my inital question now all run except for one oddball that fails in ProSdrAtmsRemapController.exe. I'm working on that one now, but it's off-topic to this thread, so I'll close this out with all my thanks.

Rich

[kbisanz]

Glad to hear you got this working.

Obviously the results you get for CrIMSS EDR may not exactly match operational results since you're using different NCEP data. But if you're just trying to get the algorithm to run to completion it's perfectly fine and results should be pretty close.

[rnc]

Hi Kevin
My intuition is that the exact NCEP interpolation interval should have a minor effect on the retrieval -- but you are right that I am ultimately interested in matching the operational code as closely as possible (especially given so much time invested). I am going to work next on using ADL to do the interpolation rather than the perl script. I could not get the landing zone technique to work earlier but I will give it another go.

BTW, that last granule I had trouble with was actually failing way back in ProSdrAtmsController, and it had to do with N_Granule_Version attribute set to "A2" rather than "A1" like every other granule I've tried to process. What is the significance of that flag, and is it OK to simply change it in the .asc file? I did that to verify it was indeed my problem and the EDR ran to completion.

Thanks

Rich

[kbisanz]

Yeah, I would recommend using ADL to ingest and interpolate the data. Please post back if you have issues you can't figure out. What version of ADL are you using? ADL 4.0? ADL 4.1?

The A1 or A2 denotes if the granule has been reprocessed due to new data. A granule that is reprocessed is set as A2. It's somewhat important because the granule version of inputs must match (unless the input is a non-center granule cross granule input). In the LW file near the bottom is a TaskDetails tag that probably states "A1". This means only A1 inputs can be used.

You can change the .asc file to contain A1 and it should work.

Similar questions with A1 or A2 have been asked a couple of times on the forum. One is viewtopic.php?f=27&t=225 .

[rnc]

Hi Kevin -

I actually asked this last week at the workshop at Greentec and based upon the answers I received, I think I am actually doing this in a legitimate way.... It seems that MSD ingest only (?) accepts HDF5 wrapped internal files, UW only serves .grib2 files, and Paul Meade's perl scripts are the "right" way to do the conversion. As it turns out, Paul's script uses ADL's ProAncTemporalInterpolation.exe routine, and I have the log file from that program so I know it's being run.

Considering this does actually work,I'm inclined to leave it alone at this point. However, if there's a straightforward way to drop grib files into the landing zone and ingest them, I'll give it a shot. I've tried a few different things but nothing seemed to be accepted.

Thanks again

Rich