CSPP VIIRS Active Fire Version 2.1 Test data can't be processed

[mbereczky]

Hello again,

I'm trying to set up CSPP VIIRS Active Fire Version 2.1 and test it with the test data, following the steps described in the CSPP_Active_Fire_Installation_Guide_v2.1.0. However, following the instructions results in an error message while processing the test data.

I tried it using DEBIAN 11 and in a Rocky Linux 8 container with the same result. No run is successful, for each run I get the following error:

Code: Select all

General warning for  J02003730587901
        Problem generating LWM for granule_id J02003730587901
list index out of range
        Ancillary granulation failed for granule_id J02003730587901
Maximum number of attempts (Land/Water Mask: J02003730586194) of obtaining return code reached for "20", setting return code to zero.
        LWM granulation for J02003730586194 took 0 minutes, 11.429446 seconds
Traceback (most recent call last):
  File "/cspp-active-fire-noaa_2.1/scripts/ancillary/stage_ancillary.py", line 175, in get_lwm
    shutil.move(glob("AF-LAND_MASK_375M*.nc")[0], lwm_file)
IndexError: list index out of range

Here's the minimal example I used to create the Rocky Linux 8 Docker image:

Code: Select all

FROM rockylinux:8

RUN dnf -y install wget

# Install CSPP Active Fire
RUN wget https://bin.ssec.wisc.edu/pub/CSPP/hidden/FIRE/v2.1/cspp-active-fire-noaa_2.1.tar.gz && \ 
    tar xf cspp-active-fire-noaa_2.1.tar.gz

# Get testdata
RUN wget https://bin.ssec.wisc.edu/pub/CSPP/hidden/FIRE/v2.1/cspp-active-fire-test_data_2.1.tar.gz && \
    tar xf cspp-active-fire-test_data_2.1.tar.gz && \
    cd fire_test_data && \
    mkdir work

You still have to set the environment variables after building and launching the container:

Code: Select all

docker build . -t cspp-fire
docker run -it --rm cspp-fire:latest

cd cspp-active-fire-noaa_2.1

export CSPP_ACTIVE_FIRE_HOME=$PWD ; cd ..
source $CSPP_ACTIVE_FIRE_HOME/cspp_active_fire_noaa_env.sh

cd ../fire_test_data/work/
cspp_active_fire_noaa.sh --num-cpu 4 --pa-dir ../PA_dir ../input

I've attached the full log in debug mode in PDF, as I can't attach any text files.

Based on the documentation, I would expect this to run without any errors. Am I missing something or is there a problem with the test data?

Best,
Max

[kathys]

Hello,

We are investigating the problem that you are having.

Kathy

[mbereczky]

Thanks!

I get the same problem when trying to run the processor with my own SDRs generated using CSPP SDR V4.0.

Can you reproduce the issue?

[kathys]

We have not experienced this problem on the target Rocky Linux 8 OS. The log file that you provided is showing a problem with the step that granulates the land/water static ancillary file to match the input SDR granules.

Not sure why a Docker Rocky 8 installation would cause a problem, but we continue to investigate.

Kathy

[mbereczky]

I got the same problem on a non-docker Debian 11.

If you can't reproduce the issue that makes me wonder if I'm setting some configuration wrongly I assume there's nothing crucial missing in the steps I described in the minimal example?

[mbereczky]

Just checked that changing the Docker image to CentOS 7.5 doesn't help

[kathys]

Sorry that you are having so much trouble successfully executing the software. Can you please download and execute the bash shell script below and send the output to us? The script executes a series of commands just to gather info about your system. It will help us to identify any system configurations that may be causing the problem.

https://bin.ssec.wisc.edu/pub/CSPP/test ... eo-diag.sh

Before execution, make sure it is executable:

chmod guo+x cspp-leo-diag.sh

Then execute:

./cspp-leo-diag.sh

Kathy

[mbereczky]

Hi Kathy,

attached is the output of the script.

In the meantime, I've set up a different VM running CentOS 7 and was able to run the test data and process our own DB data (even with the RockyLinux8 docker images). So the problem must somehow be related to the hardware/software stack on the other server (where the attached system info is from). If you need any more specifics, please let us know.

As these are our production servers, it would still be important to get the software running there.

Really appreciate your help!

Best,
Max

[kathys]

Sorry for taking so long to respond. We are having trouble determining the source of the problem that you are experiencing. Please follow the instructions below. This will help us to understand what is happening in the land/water granulation step of the retrieval process, which is where your installation is failing.

1). Do a fresh installation of the CSPP Active Fire Version 2.1 software and set the environmental variables.

2). Untar the test data again as described in Section 2.2 of the installation guide to make sure the directories are unmodified.

3). Create a new sub-directory within the test data main directory, and cd into it:

cd fire_test_data/
mkdir j02_gran
cd j02_gran

4). Create a list of the VIIRS I-Band geolocation files in the test data, which are used as input to the land/water granulation process.

ls -d ../input/GITCO*.h5 > granules_list

5). Manually execute the I-Band granulation process

$CSPP_ACTIVE_FIRE_HOME/vendor/granulation_iband_static granules_list $PWD \
$CSPP_ACTIVE_FIRE_HOME/static_ancillary/LandWater15ARC.nc

6). Can you please send us the error messages that you see when trying to execute this command?

Good luck,

Kathy

[mbereczky]

Hi Kathy,

we solved the mystery:

The problem seems to be caused by the statically linked binaries containing the vsyscall that seems to be disabled by default on Debian since version 9 (release 2017) (see https://helpcenter.onlyoffice.com/insta ... scall.aspx).

The log lines providing this hint can found in /var/log/syslog:

Code: Select all

Nov 14 17:35:37 hydra-dev-ax101-hel1 kernel: [1755451.535618] granulation_iba[284761] vsyscall attempted with vsyscall=none ip:ffffffffff600400 cs:33 sp:7ffe4e2d3b58 ax:ffffffffff600400 si:821ac0 di:0
Nov 14 17:35:37 hydra-dev-ax101-hel1 kernel: [1755451.535796] granulation_iba[284761]: segfault at ffffffffff600400 ip ffffffffff600400 sp 00007ffe4e2d3b58 error 15
Nov 14 17:35:37 hydra-dev-ax101-hel1 kernel: [1755451.535960] Code: Unable to access opcode bytes at RIP 0xffffffffff6003d6.

After enabling vsyscall as a kernel boot parameter (as described in the previously linked help center article) the program finally runs through on our Debian machines using the CentOS 7.9 docker image.

Thank you very much for your kind support with this issue, highly appreciated!

Best,
Max