SSEC forums

discussion forums for the Space Science & Engineering Center at UW-Madison
https://forums.ssec.wisc.edu/

GAASP Test Files

https://forums.ssec.wisc.edu/viewtopic.php?t=59306

Page 1 of 1

GAASP Test Files

Posted: Tue Nov 16, 2021 4:25 am
by rdirienzo
Hello everyone and thanks for your help.

I have a problem with the test files:

after installing https://bin.ssec.wisc.edu/pub/CSPP/hidd ... 1.0.tar.gz
and downloaded https://bin.ssec.wisc.edu/pub/CSPP/hidd ... 1.0.tar.gz

I ran the test suggested by the manual

cd cspp-gaasp-test_data-1.0/
mkdir work
cd work
cspp_gaasp.sh --num-cpu 5 -i ../inputs

with this result:

Code: Select all

################################################################################
#                    Community Satellite Processing Package                    #
#               GCOM-W1 AMSR2 Algorithm Software Package (GAASP)               #
#                                                                              #
#                               CSPP GAASP: 1.0                                #
#                                 GAASP: v2r2                                  #
#                Current date: 2021-11-16 09:00:01.869886+00:00                #
################################################################################

Using cache dir /opt/test/cspp-gaasp-test_data-1.0/work/cspp_gaasp_cache_dir

>>> Input Files

    <Granule ID>         <Granule Start Time>
 202003052342_180A       2020-03-05 23:42:00
 202003060951_043D       2020-03-06 09:51:00
 202003061128_059D       2020-03-06 11:28:00
 202003061306_075D       2020-03-06 13:06:00
 202003061444_091D       2020-03-06 14:44:00


>>> Running GAASP...

There are 8 available CPUs
        We have requested 5 CPUs
        We are using 5/8 available CPUs

Running GAASP algs: preprocessor, ocean, precipitation, snow, soil, seaice, postprocessor

>>> Processing granule ID 202003052342_180A...

        >>> Running Preprocessor...
        *[ERROR]: GAASP Preprocessor module has failed to ingest Sea Ice ancillary data.
                  no sea ice files are available for 2020-03-05 23:42:00

        >>> GAASP execution of 202003052342_180A took 0 days, 0 hours, 0 minutes, 13.538677 seconds


>>> Processing granule ID 202003060951_043D...

        >>> Running Preprocessor...
        *[ERROR]: GAASP Preprocessor module has failed to ingest Sea Ice ancillary data.
                  no sea ice files are available for 2020-03-06 09:51:00

        >>> GAASP execution of 202003060951_043D took 0 days, 0 hours, 0 minutes, 3.434243 seconds


>>> Processing granule ID 202003061128_059D...

        >>> Running Preprocessor...
        *[ERROR]: GAASP Preprocessor module has failed to ingest Sea Ice ancillary data.
                  no sea ice files are available for 2020-03-06 11:28:00

        >>> GAASP execution of 202003061128_059D took 0 days, 0 hours, 0 minutes, 3.515479 seconds


>>> Processing granule ID 202003061306_075D...

        >>> Running Preprocessor...
        *[ERROR]: GAASP Preprocessor module has failed to ingest Sea Ice ancillary data.
                  no sea ice files are available for 2020-03-06 13:06:00

        >>> GAASP execution of 202003061306_075D took 0 days, 0 hours, 0 minutes, 3.425944 seconds


>>> Processing granule ID 202003061444_091D...

        >>> Running Preprocessor...
        *[ERROR]: GAASP Preprocessor module has failed to ingest Sea Ice ancillary data.
                  no sea ice files are available for 2020-03-06 14:44:00

        >>> GAASP execution of 202003061444_091D took 0 days, 0 hours, 0 minutes, 3.497752 seconds


>>> GAASP output summary...

    <Granule ID>    |  preprocessor       ocean      precipitation      snow           soil          seaice      postprocessor
 202003052342_180A  |        o              o              o              o              o              o              o
 202003060951_043D  |        o              o              o              o              o              o              o
 202003061128_059D  |        o              o              o              o              o              o              o
 202003061306_075D  |        o              o              o              o              o              o              o
 202003061444_091D  |        o              o              o              o              o              o              o
Do I need to run in a different way or do I need a different dataset?

Code: Select all

 find work
.
./202003060951_043D
./202003060951_043D/preprocessor
./202003061128_059D
./202003061128_059D/preprocessor
./202003061306_075D
./202003061306_075D/preprocessor
./202003052342_180A
./202003052342_180A/preprocessor
./cspp_gaasp_cache_dir
./202003061444_091D
./202003061444_091D/preprocessor
./cspp_gaasp.20211116T090001.log

Code: Select all

XDG_SESSION_ID=307
HOSTNAME=toolboxes
SELINUX_ROLE_REQUESTED=
TERM=xterm-256color
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=195.81.105.197 55271 22
SELINUX_USE_CURRENT_RANGE=
OLDPWD=/opt/test/cspp-gaasp-test_data-1.0/work/cspp_gaasp_cache_dir
SSH_TTY=/dev/pts/0
CSPP_GAASP_HOME=/opt/CSPP/cspp-gaasp-1.0
USER=root
LD_LIBRARY_PATH=:/common/local/lib/
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
MAIL=/var/spool/mail/root
PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/go/bin:/root/bin:/opt/CSPP/cspp-gaasp-1.0/vendor/ShellB3/bin:/opt/CSPP/cspp-gaasp-1.0/bin:/opt/CSPP/cspp-gaasp-1.0/scripts
PWD=/opt/test/cspp-gaasp-test_data-1.0/work
LANG=en_US.UTF-8
SELINUX_LEVEL_REQUESTED=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=195.81.105.197 55271 65.21.187.23 22
GOPATH=/root/go
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/0
_=/usr/bin/env

Code: Select all

top - 09:13:02 up 10 days, 15:18,  1 user,  load average: 0.00, 0.01, 0.05
Tasks: 120 total,   2 running, 118 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32005448 total,  2701344 free,   421768 used, 28882336 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 30904764 avail Mem
Thanks for your help and time,
Roberto

Re: GAASP Test Files

Posted: Tue Nov 16, 2021 11:18 am
by kathys
Ciao Roberto,

The problem is that the ancillary data required to execute the test data was accidentally removed from the external CSPP ancillary data archive. I have now replaced the data for the test data date, 6 March 2020 on the website.

Please try executing the GAASP software on the test data again, and we apologize for this error.

Kathy

Re: GAASP Test Files

Posted: Thu Nov 18, 2021 4:56 am
by rdirienzo
Thanks for your help!
The results of the new tests:

Code: Select all

################################################################################
#                    Community Satellite Processing Package                    #
#               GCOM-W1 AMSR2 Algorithm Software Package (GAASP)               #
#                                                                              #
#                               CSPP GAASP: 1.0                                #
#                                 GAASP: v2r2                                  #
#                Current date: 2021-11-18 09:21:03.585310+00:00                #
################################################################################

Using cache dir /opt/test/cspp-gaasp-test_data-1.0/work/cspp_gaasp_cache_dir

>>> Input Files

    <Granule ID>         <Granule Start Time>
 202003052342_180A       2020-03-05 23:42:00
 202003060951_043D       2020-03-06 09:51:00
 202003061128_059D       2020-03-06 11:28:00
 202003061306_075D       2020-03-06 13:06:00
 202003061444_091D       2020-03-06 14:44:00


>>> Running GAASP...

There are 8 available CPUs
        We have requested 5 CPUs
        We are using 5/8 available CPUs

Running GAASP algs: preprocessor, ocean, precipitation, snow, soil, seaice, postprocessor

>>> Processing granule ID 202003052342_180A...

        >>> Running Preprocessor...
        *[ERROR]: GAASP Preprocessor module has failed to ingest Sea Ice ancillary data.
                  no sea ice files are available for 2020-03-05 23:42:00

        >>> GAASP execution of 202003052342_180A took 0 days, 0 hours, 0 minutes, 8.401295 seconds


>>> Processing granule ID 202003060951_043D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 5.677264 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 1.249006 seconds
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 3.069922 seconds
        *[ERROR]: GAASP Ocean module has failed to ingest GFS ancillary data.
                  Could not retrieve forecast files for 2020-03-05 21:51:00!
Tried: ['https://jpssdb.ssec.wisc.edu/cspp_v_2_0/ancillary/2020_03_05_065/gfs.t18z.pgrb2.0p50.f012.20200305', 'https://jpssdb.ssec.wisc.edu/cspp_v_2_0/ancillary/2020_03_06_066/gfs.t00z.pgrb2.0p50.f012.20200306']
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 18.271762 seconds
        GAASP Snow          took 0 days, 0 hours, 5 minutes, 1.707545 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 3.092556 seconds

        >>> GAASP execution of 202003060951_043D took 0 days, 0 hours, 5 minutes, 14.484129 seconds


>>> Processing granule ID 202003061128_059D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 7.450701 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 1.576741 seconds
        *[ERROR]: GAASP Ocean module has failed to ingest GFS ancillary data.
                  Could not retrieve forecast files for 2020-03-05 23:28:00!
Tried: ['https://jpssdb.ssec.wisc.edu/cspp_v_2_0/ancillary/2020_03_05_065/gfs.t18z.pgrb2.0p50.f012.20200305', 'https://jpssdb.ssec.wisc.edu/cspp_v_2_0/ancillary/2020_03_06_066/gfs.t00z.pgrb2.0p50.f012.20200306']
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 2.864369 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 46.126937 seconds
        GAASP Snow          took 0 days, 0 hours, 6 minutes, 45.012709 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 3.382884 seconds

        >>> GAASP execution of 202003061128_059D took 0 days, 0 hours, 6 minutes, 55.942108 seconds


>>> Processing granule ID 202003061306_075D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 7.415276 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 2.705065 seconds
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 3.080443 seconds
        GAASP Ocean         took 0 days, 0 hours, 0 minutes, 13.332563 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 43.610499 seconds
        GAASP Snow          took 0 days, 0 hours, 7 minutes, 4.923519 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 4.632866 seconds

        >>> GAASP execution of 202003061306_075D took 0 days, 0 hours, 7 minutes, 17.050406 seconds


>>> Processing granule ID 202003061444_091D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 6.407116 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 3.384934 seconds
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 3.924118 seconds
        GAASP Ocean         took 0 days, 0 hours, 0 minutes, 12.339296 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 23.011164 seconds
        GAASP Snow          took 0 days, 0 hours, 9 minutes, 48.796337 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 4.498803 seconds

        >>> GAASP execution of 202003061444_091D took 0 days, 0 hours, 9 minutes, 59.788125 seconds


>>> GAASP output summary...

    <Granule ID>    |  preprocessor       ocean      precipitation      snow           soil          seaice      postprocessor
 202003052342_180A  |        o              o              o              o              o              o              o
 202003060951_043D  |        +              o              +              +              +              +              +
 202003061128_059D  |        +              o              +              +              +              +              +
 202003061306_075D  |        +              +              +              +              +              +              +
 202003061444_091D  |        +              +              +              +              +              +              +

Re: GAASP Test Files

Posted: Thu Nov 18, 2021 10:48 am
by kathys
Sorry Roberto, I needed to replace the ancillary data on the CSPP ancillary website from 5 March 2020 too. I have done that. Please try executing the software using the test data again.

Kathy

Re: GAASP Test Files

Posted: Mon Nov 22, 2021 6:47 am
by rdirienzo
Ciao Kathy,
Thank you for your time.
With your help the test was successful!!

Thanks again,
Roberto

New results:

Code: Select all

################################################################################
#                    Community Satellite Processing Package                    #
#               GCOM-W1 AMSR2 Algorithm Software Package (GAASP)               #
#                                                                              #
#                               CSPP GAASP: 1.0                                #
#                                 GAASP: v2r2                                  #
#                Current date: 2021-11-22 11:01:27.053094+00:00                #
################################################################################

Using cache dir /opt/cspp-gaasp-test_data-1.0/work/cspp_gaasp_cache_dir

>>> Input Files

    <Granule ID>         <Granule Start Time>
 202003052342_180A       2020-03-05 23:42:00
 202003060951_043D       2020-03-06 09:51:00
 202003061128_059D       2020-03-06 11:28:00
 202003061306_075D       2020-03-06 13:06:00
 202003061444_091D       2020-03-06 14:44:00


>>> Running GAASP...

There are 8 available CPUs
        We have requested 5 CPUs
        We are using 5/8 available CPUs

Running GAASP algs: preprocessor, ocean, precipitation, snow, soil, seaice, postprocessor

>>> Processing granule ID 202003052342_180A...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 7.821300 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 2.712384 seconds
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 2.741771 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 23.294708 seconds
        GAASP Ocean         took 0 days, 0 hours, 0 minutes, 13.610108 seconds
        GAASP Snow          took 0 days, 0 hours, 10 minutes, 20.686978 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 4.553532 seconds

        >>> GAASP execution of 202003052342_180A took 0 days, 0 hours, 10 minutes, 40.014123 seconds


>>> Processing granule ID 202003060951_043D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 5.327671 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 2.813249 seconds
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 3.045970 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 14.112322 seconds
        *[WARNING]: GAASP Ocean binary had a non-fatal error (GAASP return code 1).
                    Please examine /opt/cspp-gaasp-test_data-1.0/work/202003060951_043D/ocean/*.log
        GAASP Snow          took 0 days, 0 hours, 4 minutes, 43.521562 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 3.070814 seconds

        >>> GAASP execution of 202003060951_043D took 0 days, 0 hours, 4 minutes, 56.527640 seconds


>>> Processing granule ID 202003061128_059D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 7.547461 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 2.796605 seconds
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 3.369402 seconds
        GAASP Ocean         took 0 days, 0 hours, 0 minutes, 12.790176 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 45.477166 seconds
        GAASP Snow          took 0 days, 0 hours, 6 minutes, 29.469117 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 4.393639 seconds

        >>> GAASP execution of 202003061128_059D took 0 days, 0 hours, 6 minutes, 41.498540 seconds


>>> Processing granule ID 202003061306_075D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 7.628820 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 2.955748 seconds
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 4.421306 seconds
        GAASP Ocean         took 0 days, 0 hours, 0 minutes, 13.161544 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 42.825208 seconds
        GAASP Snow          took 0 days, 0 hours, 6 minutes, 44.767944 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 4.488783 seconds

        >>> GAASP execution of 202003061306_075D took 0 days, 0 hours, 6 minutes, 56.982833 seconds


>>> Processing granule ID 202003061444_091D...

        >>> Running Preprocessor...
        GAASP Preprocessor took 0 days, 0 hours, 0 minutes, 6.356874 seconds

        >>> Running Science Algorithms...
        Submitting 5 GAASP science tasks to the pool...
        GAASP Soil          took 0 days, 0 hours, 0 minutes, 2.832565 seconds
        GAASP Seaice        took 0 days, 0 hours, 0 minutes, 5.762341 seconds
        GAASP Ocean         took 0 days, 0 hours, 0 minutes, 11.581830 seconds
        GAASP Precipitation took 0 days, 0 hours, 0 minutes, 21.658089 seconds



        GAASP Snow          took 0 days, 0 hours, 9 minutes, 6.355029 seconds

        >>> Running Postprocessor...
        GAASP Postprocessor took 0 days, 0 hours, 0 minutes, 4.339823 seconds

        >>> GAASP execution of 202003061444_091D took 0 days, 0 hours, 9 minutes, 17.134741 seconds


>>> GAASP output summary...

    <Granule ID>    |  preprocessor       ocean      precipitation      snow           soil          seaice      postprocessor
 202003052342_180A  |        +              +              +              +              +              +              +
 202003060951_043D  |        +              o              +              +              +              +              +
 202003061128_059D  |        +              +              +              +              +              +              +
 202003061306_075D  |        +              +              +              +              +              +              +
 202003061444_091D  |        +              +              +              +              +              +              +
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited