Problem with NCI new Thredds data service

[laetis]

Hello,

I am using geocat software to to convert himawari9 data to netcdf before doing a WRFDA run.
I am getting the himawari 9 data fom NCI (https://opus.nci.org.au/spaces/NDP/page ... DS+Upgrade). Recently they upgrade their services. Since then, I have WRFDA runs ending with segmentation fault at the moment of reading the radiance data. I am wondering if , maybe, geocat software would not be compatible with these new data release? On their website there is this obscure (to me) comment:
"Unidata has changed the THREDDS NetCDF Subset Service (NCSS) output in the new version. This can result in differences in the NCSS dimension ordering. For example, the lat/long values can display in a different order in the output. The issue has been reported to Unidata and remains under development."
Could that be a problem?
I should add that when running geocat, I don't have an error values and the netcdf files are generated apparently without problem. But I was wondering if it could be somehow, corrupted?

Best

[geoffc]

Thanks for your question about the NCI Himawari 9 data and Geocat. I just want to confirm, the files you are obtaining from NCI are of the form "HS_H09_20170308_0230_B15_FLDK_R20_S0310.DAT" etc..., and you are having trouble running these files from the new server through geocat, as opposed to from the now-deprecated server?

I will try to obtain some data from NCI and replicate your error, and may have followup questions.

Thanks,
Geoff

[laetis]

Thank you for your answer.
Yes the files are of this type exactly.
And prior to the change of server by NCI (my previous runs are about 1 year ago), I was never having issues.
After running geocat I don't see any problem but strangely WRFDA gives a segmentation fault. Since nothing has changed except the migration/update of nci...

[geoffc]

Do you have access to samples of the H9 data before and after the update? If yes, could you try and upload a tarball of the data to ftp.ssec.wisc.edu:/pub/incoming ?

Thanks,
Geoff

[laetis]

Yes I have old data.
I created a tarball but failed to push it:
curl -T RADIANCE_DATA_20240428_0600.tar.bz2 ftp://ftp.ssec.wisc.edu:/pub/incoming
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
curl: (25) Failed FTP upload: 550

[laetis]

I upload a tarball of old files to the ftp server.
The name of the tarball is: RADIANCE_DATA_20240428_0600.tar.bz2

Thanks for your help.

Best

[geoffc]

Can you check the file `DATA_20240428_0600/HS_H09_20240428_0600_B09_FLDK_R20_S0410.DAT` does it have the same size as the others? It may have been corrupted.

[laetis]

Hi,
Maybe a problem with the tarball because it is ok . I pushed it alone on the server.

[geoffc]

When I try to unpack the tarball I get an error about the tarball being truncated. I ran

Code: Select all

bunzip2 -tvvf RADIANCE_DATA_20240428_0600.tar.bz2

to check the integrity of the tarball and I got

Code: Select all

    ...
    ...
    [2670: huff+mtf rt+rld]
    [2671: huff+mtf rt+rld]
    [2672: huff+mtf rt+rld]
    [2673: huff+mtf file ends unexpectedly
    
You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Can you run the bunzip2 command I gave above on your local copy of the tarball and see what it says. Also, what is the size of the tarball on your machine?

[geoffc]

After having a better look, I am missing the files HS_H09_20240428_0600_B08_FLDK_R20_S1010.DAT and HS_H09_20240428_0600_B09_FLDK_R20_S0810.DAT, could you post those files individually?