WikiPrint - from Polar Technologies

WRF4G Tutorial part 2

How to manage WRF4G errors

In this section, we are going to see how to manage WRF4G errors. In order to do that, we are going to create a new experiment called test_1, based on single_test, in which the end_date will be "2011-08-30_12:00:00". Follow the steps below.

[user@mycomputer~]$ cd $WRF4G_LOCATION/experiments

[user@mycomputer~]$ ls
single_test  wrfuc_physics  wrfuc_single_serial

[user@mycomputer~]$ cp -r single_test single_test_1

[user@mycomputer~]$ cd single_test_1

[user@mycomputer~]$ cat experiment.wrf4g | grep "experiment_name"
experiment_name = "test"

[user@mycomputer~]$ cat experiment.wrf4g | grep "start_date="
start_date="2011-08-28_12:00:00"

[user@mycomputer~]$ cat experiment.wrf4g | grep "experiment_name"
experiment_name = "test_1"

[user@mycomputer~]$ cat experiment.wrf4g | grep "start_date "
start_date="2011-08-28_00:00:00"

[user@mycomputer~]$ wrf4g_prepare
Warning: You are using resources.wrf4g located in the /home/carlos/WRF4G/etc/ directory.
Preparing namelist...
WRFV3/run/namelist.input
WRF Check Warning: CAM radiation selected but paerlev/levsiz/cam_abs_dim1/cam_abs_dim2 was not set. Fixing...
WRF Check Warning: radt is shorter than dx (0.500000)

---> Single params run
---> Continuous run
        ---> cycle_chunks: test_1 2011-08-28_00:00:00 2011-08-30_00:00:00
                ---> chunks 1: test_1 2011-08-28_00:00:00 2011-08-28_12:00:00
                ---> chunks 2: test_1 2011-08-28_12:00:00 2011-08-29_00:00:00
                ---> chunks 3: test_1 2011-08-29_00:00:00 2011-08-29_12:00:00
                ---> chunks 4: test_1 2011-08-29_12:00:00 2011-08-30_00:00:00

[user@mycomputer~]$ wrf4g_status --long 
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
test                 2     D  3/3    mycomputer ciclon     Finished       0 100.00
test_1               -     P  0/4    -          -          Prepared       - 0.00

[user@mycomputer~]$ wrf4g_submit
Submitting realization: "test_1"
        Submitting Chunk 1:     2011-08-28_00:00:00     2011-08-28_12:00:00
        Submitting Chunk 2:     2011-08-28_12:00:00     2011-08-29_00:00:00
        Submitting Chunk 3:     2011-08-29_00:00:00     2011-08-29_12:00:00
        Submitting Chunk 4:     2011-08-29_12:00:00     2011-08-30_00:00:00

[user@mycomputer~]$ wrf4g_status --long
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
test                 2     D  3/3    mycomputer ciclon     Finished       0 100.00
test_1               3     F  1/4    mycomputer ciclon     Failed        62 0.00

Like you can see before, the realization test_1 has finished with an exit code 62. What happened ? The exit code 62 indicates that ungrib binary had an error during its execution. In order to solve the error, we are going to check out the log of the chunk number 1.

[user@mycomputer~]$ cat $WRF4G_LOCATION/etc/resources.wrf4g | grep WRF4G_BASEPATH=
WRF4G_BASEPATH="/home/user/WRF4G/repository/output"

[user@mycomputer~]$ cd $WRF4G_LOCATION/repository/output/test_1/test_1/log/

[user@mycomputer~]$ ls
log_1_4.tar.gz 

The chunk log name is composed of using chunk number and job identifier (GW).

In our case, chunk log name will be log_1_4.tar.gz because the chunk number is 1 and the job identifier is 4.

[user@mycomputer~]$ tar xzvf log_1_4.tar.gz 
WRF4G.log
configure.wps
ls.wps
ls.wrf
ungrib_GFS_2011082800.out

In each log package you are able to see all WRF log binaries as well as WRF4G logs such as WRF4G.log, wrfgel.out and monitor.log. In our case, we are going to focus on WRF4G.log which is the main log.

[user@mycomputer~]$ cat WRF4G.log
* Mon Oct  1 17:27:45 CEST 2012: Creating WRF4G structure ... 
`/home/user/WRF4G/repository/apps/WRFbin-3.1.1_r832INTEL_OMPI.tar.gz' -> `/home/user/.gw_user_3/WRFbin-3.1.1_r832INTEL_OMPI.tar.gz'
`/home/user/WRF4G/repository/output/test_1/test_1/namelist.input' -> `/home/user/.gw_user_3/WRFV3/run/namelist.input'
* Mon Oct  1 17:27:46 CEST 2012: Preparing WRF4G binaries ... 
* Mon Oct  1 17:27:46 CEST 2012: Creating parallel environment ... 
* Mon Oct  1 17:27:46 CEST 2012: Using default configuration ... 
* Mon Oct  1 17:27:46 CEST 2012: Checking restart information ... 
* Mon Oct  1 17:27:46 CEST 2012: The boundaries and initial conditions are not available ... 
* Mon Oct  1 17:27:46 CEST 2012: Downloading geo_em files and namelist.wps ... 
/home/user/.gw_user_3/WRFGEL/vcp -v /home/user/WRF4G/repository/domains/Santander_50km/* .
cp -v -R /home/user/WRF4G/repository/domains/Santander_50km/* /home/user/.gw_user_3/WPS
`/home/user/WRF4G/repository/domains/Santander_50km/geo_em.d01.nc' -> `/home/user/.gw_user_3/WPS/geo_em.d01.nc'
`/home/user/WRF4G/repository/domains/Santander_50km/namelist.wps' -> `/home/user/.gw_user_3/WPS/namelist.wps'
* Mon Oct  1 17:27:46 CEST 2012: Modifying namelist ... 
Updating parameter start_date in file: namelist.wps
Updating parameter end_date in file: namelist.wps
Updating parameter max_dom in file: namelist.wps
Updating parameter prefix in file: namelist.wps
Updating parameter interval_seconds in file: namelist.wps
* Mon Oct  1 17:27:46 CEST 2012: About to run preprocessor and Ungrib ... 
* Mon Oct  1 17:27:46 CEST 2012: Running preprocessor.default ... 
Linking global data from: /home/user/WRF4G/repository/input/NCEP/GFS
`/home/user/.gw_user_3/WPS/grbData/gfs2011082812_00.grb' -> `/home/user/WRF4G/repository/input/NCEP/GFS/2011/gfs2011082812_00.grb'
`/home/user/.gw_user_3/WPS/grbData/gfs2011082812_06.grb' -> `/home/user/WRF4G/repository/input/NCEP/GFS/2011/gfs2011082812_06.grb'
`/home/user/.gw_user_3/WPS/grbData/gfs2011082812_12.grb' -> `/home/user/WRF4G/repository/input/NCEP/GFS/2011/gfs2011082812_12.grb'
`/home/user/.gw_user_3/WPS/grbData/gfs2011082812_18.grb' -> `/home/user/WRF4G/repository/input/NCEP/GFS/2011/gfs2011082812_18.grb'
`/home/user/.gw_user_3/WPS/grbData/gfs2011082812_24.grb' -> `/home/user/WRF4G/repository/input/NCEP/GFS/2011/gfs2011082812_24.grb'
`/home/user/.gw_user_3/WPS/grbData/gfs2011082812_30.grb' -> `/home/user/WRF4G/repository/input/NCEP/GFS/2011/gfs2011082812_30.grb'
`/home/user/.gw_user_3/WPS/grbData/gfs2011082812_36.grb' -> `/home/user/WRF4G/repository/input/NCEP/GFS/2011/gfs2011082812_36.grb'
* Mon Oct  1 17:27:47 CEST 2012: Running ungrib ... 
**********************************************************************************
WRF4G was deployed in ... 
    /home/user/.gw_user_3
and it ran in ...
    /home/user/.gw_user_3
**********************************************************************************

WRF4G.log shows that ungrib was the last WRF binary. Therefore, if you check out ungrib log, you will probably discover the error.

[user@mycomputer~]$ tail ungrib_GFS_2011082800.out
 200.0  X        X        X        X        X                                                                                                                 
 150.0  X        X        X        X        X                                                                                                                 
 100.0  X        X        X        X        X                                                                                                                 
  70.0  X        X        X        X        X                                                                                                                 
  50.0  X        X        X        X        X
                                                                         
Subroutine DATINT: Interpolating 3-d files to fill in any missing data...
Looking for data at time 2011-08-28_00
ERROR: Data not found: 2011-08-28_00:00:00.0000
 Begin rrpr

-------------------------------------------------------------------------------

The problem is that there is not input data to simulate the last chunk.

[user@mycomputer~]$cat experiment.wrf4g | grep "extdata_path"
extdata_path = "${WRF4G_INPUT}/NCEP/GFS"

[user@mycomputer~]$ cat $WRF4G_LOCATION/etc/resources.wrf4g | grep "WRF4G_INPUT="
WRF4G_INPUT="/home/user/WRF4G/repository/input"

[user@mycomputer~]$ ls -l /home/user/WRF4G/repository/input/NCEP/GFS/2011/
total 36388
-rw-r--r-- 1 user user 4909850 2012-09-13 11:38 gfs2011082812_00.grb
-rw-r--r-- 1 user user 5411705 2012-09-13 11:38 gfs2011082812_06.grb
-rw-r--r-- 1 user user 5411214 2012-09-13 11:38 gfs2011082812_12.grb
-rw-r--r-- 1 user user 5415031 2012-09-13 11:38 gfs2011082812_18.grb
-rw-r--r-- 1 user user 5397677 2012-09-13 11:38 gfs2011082812_24.grb
-rw-r--r-- 1 user user 5386190 2012-09-13 11:38 gfs2011082812_30.grb
-rw-r--r-- 1 user user 5316014 2012-09-13 11:38 gfs2011082812_36.grb

Now, create a new experiment based on single_test then add this line timestep_dxfactor="manual:10000" to experiment.wrf4g file as well as change the experiment_name.

How to add new computing resources to WRF4G

WRF4G uses DRM4G to access to different Distributed Resource Managements (DRM) such as:

In order to add new resources, you need to edit Computing Resources section in the framework4g.conf file which is located under $WRF4G_LOCATION/etc directory. The file has to contain one resource per line with the format:

   resource_name   attributes
   ...     ...
   resource_name   attributes

where:

Variable options:

Examples of configuration:

mycomputer    local://localhost?LRMS_TYPE=fork;NODECOUNT=2
PBS_cluster   local://localhost?LRMS_TYPE=pbs;QUEUE_NAME=estadistica
SGE_cluster   local://localhost?LRMS_TYPE=sge;PROJECT=l.project

If you want to configure a remote computing resource through ssh protocol, you need to put your private keys into your ?ssh-agent, and it will handle your authentication thereafter (see Appendix A) or set up SSH login without password (see Appendix B).

In addition, you will probably need to update WRF4G_BASEPATH, WRF4G_DOMAINPATH, WRF4G_INPUT and WRF4G_APPS variables, which are defined in resources.wrf4g . Due to the fact that these variables may point to other machines. See running environment for more information.

remote_PBS    ssh://user@meteo1.macc.unican.es?LRMS_TYPE=pbs;QUEUE_NAME=short

After modifying Computing Resources section, in order to make changes effective, you have to execute:
wrf4g_framework reload

Now, try to configure this remote resource via ssh configuring framework4g.conf:
SMG_PBS ssh://coursexx@meteo1.macc.unican.es?LRMS_TYPE=pbs;QUEUE_NAME=course;NODECOUNT=2

How to use wrf4g_kill command

In this example, we are going to simulate an experiment with independent realizations which has multiple_parameters flag activated. The experiment is compose of five realizations with three chunk per realization. In order to use wrf4g_kill command, we are going to first submit the experiment.

[user@mycomputer~]$ cd $WRF4G_LOCATION/experiments/wrfuc_physics

[user@mycomputer~]$ ls
experiment.wrf4g

[user@mycomputer~]$ cat experiment.wrf4g | grep "param"
multiple_parameters=1
  multiparams_variables="mp_physics,cu_physics,ra_lw_physics,ra_sw_physics,sf_sfclay_physics,bl_pbl_physics,sf_surface_physics"
  multiparams_nitems="${max_dom},${max_dom},${max_dom},${max_dom},${max_dom},${max_dom},${max_dom}"
  multiparams_combinations="5,1:1:0,1,1,2,2,2/4,1:1:0,1,1,1,1,2/4,1:1:0,1,1,2,2,2/4,1:1:0,1,1,7,7,2/4,3:3:0,1,1,7,7,2  "
  multiparams_labels="phys1/phys2/phys3/phys4/phys5"

[user@mycomputer~]$ wrf4g_prepare 
Warning: You are using resources.wrf4g located in the /home/carlos/WRF4G/etc/ directory.
Preparing namelist...
WRFV3/run/namelist.input
WRF Check Warning: CAM radiation selected but paerlev/levsiz/cam_abs_dim1/cam_abs_dim2 was not set. Fixing...
WRF Check Warning: radt is shorter than dx (0.500000)

--->Realization: multiparams=phys1 2011-08-28_12:00:00 2011-08-30_00:00:00
Updating parameter mp_physics in file: namelist.input
Updating parameter cu_physics in file: namelist.input
Updating parameter ra_lw_physics in file: namelist.input
Updating parameter ra_sw_physics in file: namelist.input
Updating parameter sf_sfclay_physics in file: namelist.input
Updating parameter bl_pbl_physics in file: namelist.input
Updating parameter sf_surface_physics in file: namelist.input
---> Continuous run
        ---> cycle_chunks: uc_phys__phys1 2011-08-28_12:00:00 2011-08-30_00:00:00
                ---> chunks 1: uc_phys__phys1 2011-08-28_12:00:00 2011-08-29_00:00:00
                ---> chunks 2: uc_phys__phys1 2011-08-29_00:00:00 2011-08-29_12:00:00
                ---> chunks 3: uc_phys__phys1 2011-08-29_12:00:00 2011-08-30_00:00:00

--->Realization: multiparams=phys2 2011-08-28_12:00:00 2011-08-30_00:00:00
Updating parameter mp_physics in file: namelist.input
Updating parameter cu_physics in file: namelist.input
Updating parameter ra_lw_physics in file: namelist.input
Updating parameter ra_sw_physics in file: namelist.input
Updating parameter sf_sfclay_physics in file: namelist.input
Updating parameter bl_pbl_physics in file: namelist.input
Updating parameter sf_surface_physics in file: namelist.input
---> Continuous run
        ---> cycle_chunks: uc_phys__phys2 2011-08-28_12:00:00 2011-08-30_00:00:00
                ---> chunks 1: uc_phys__phys2 2011-08-28_12:00:00 2011-08-29_00:00:00
                ---> chunks 2: uc_phys__phys2 2011-08-29_00:00:00 2011-08-29_12:00:00
                ---> chunks 3: uc_phys__phys2 2011-08-29_12:00:00 2011-08-30_00:00:00

--->Realization: multiparams=phys3 2011-08-28_12:00:00 2011-08-30_00:00:00
Updating parameter mp_physics in file: namelist.input
Updating parameter cu_physics in file: namelist.input
Updating parameter ra_lw_physics in file: namelist.input
Updating parameter ra_sw_physics in file: namelist.input
Updating parameter sf_sfclay_physics in file: namelist.input
Updating parameter bl_pbl_physics in file: namelist.input
Updating parameter sf_surface_physics in file: namelist.input
---> Continuous run
        ---> cycle_chunks: uc_phys__phys3 2011-08-28_12:00:00 2011-08-30_00:00:00
                ---> chunks 1: uc_phys__phys3 2011-08-28_12:00:00 2011-08-29_00:00:00
                ---> chunks 2: uc_phys__phys3 2011-08-29_00:00:00 2011-08-29_12:00:00
                ---> chunks 3: uc_phys__phys3 2011-08-29_12:00:00 2011-08-30_00:00:00

--->Realization: multiparams=phys4 2011-08-28_12:00:00 2011-08-30_00:00:00
Updating parameter mp_physics in file: namelist.input
Updating parameter cu_physics in file: namelist.input
Updating parameter ra_lw_physics in file: namelist.input
Updating parameter ra_sw_physics in file: namelist.input
Updating parameter sf_sfclay_physics in file: namelist.input
Updating parameter bl_pbl_physics in file: namelist.input
Updating parameter sf_surface_physics in file: namelist.input
---> Continuous run
        ---> cycle_chunks: uc_phys__phys4 2011-08-28_12:00:00 2011-08-30_00:00:00
                ---> chunks 1: uc_phys__phys4 2011-08-28_12:00:00 2011-08-29_00:00:00
                ---> chunks 2: uc_phys__phys4 2011-08-29_00:00:00 2011-08-29_12:00:00
                ---> chunks 3: uc_phys__phys4 2011-08-29_12:00:00 2011-08-30_00:00:00

--->Realization: multiparams=phys5 2011-08-28_12:00:00 2011-08-30_00:00:00
Updating parameter mp_physics in file: namelist.input
Updating parameter cu_physics in file: namelist.input
Updating parameter ra_lw_physics in file: namelist.input
Updating parameter ra_sw_physics in file: namelist.input
Updating parameter sf_sfclay_physics in file: namelist.input
Updating parameter bl_pbl_physics in file: namelist.input
Updating parameter sf_surface_physics in file: namelist.input
---> Continuous run
        ---> cycle_chunks: uc_phys__phys5 2011-08-28_12:00:00 2011-08-30_00:00:00
                ---> chunks 1: uc_phys__phys5 2011-08-28_12:00:00 2011-08-29_00:00:00
                ---> chunks 2: uc_phys__phys5 2011-08-29_00:00:00 2011-08-29_12:00:00
                ---> chunks 3: uc_phys__phys5 2011-08-29_12:00:00 2011-08-30_00:00:00

[user@mycomputer~]$ wrf4g_status --long
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
uc_phys__phys1       -     P  0/3    -          -          Prepared       - 0.00
uc_phys__phys2       -     P  0/3    -          -          Prepared       - 0.00
uc_phys__phys3       -     P  0/3    -          -          Prepared       - 0.00
uc_phys__phys4       -     P  0/3    -          -          Prepared       - 0.00
uc_phys__phys5       -     P  0/3    -          -          Prepared       - 0.00

[user@mycomputer~]$ wrf4g_submit 
Submitting realization: "uc_phys__phys1"
        Submitting Chunk 1:     2011-08-28_12:00:00     2011-08-29_00:00:00
        Submitting Chunk 2:     2011-08-29_00:00:00     2011-08-29_12:00:00
        Submitting Chunk 3:     2011-08-29_12:00:00     2011-08-30_00:00:00
Submitting realization: "uc_phys__phys2"
        Submitting Chunk 1:     2011-08-28_12:00:00     2011-08-29_00:00:00
        Submitting Chunk 2:     2011-08-29_00:00:00     2011-08-29_12:00:00
        Submitting Chunk 3:     2011-08-29_12:00:00     2011-08-30_00:00:00
Submitting realization: "uc_phys__phys3"
        Submitting Chunk 1:     2011-08-28_12:00:00     2011-08-29_00:00:00
        Submitting Chunk 2:     2011-08-29_00:00:00     2011-08-29_12:00:00
        Submitting Chunk 3:     2011-08-29_12:00:00     2011-08-30_00:00:00
Submitting realization: "uc_phys__phys4"
        Submitting Chunk 1:     2011-08-28_12:00:00     2011-08-29_00:00:00
        Submitting Chunk 2:     2011-08-29_00:00:00     2011-08-29_12:00:00
        Submitting Chunk 3:     2011-08-29_12:00:00     2011-08-30_00:00:00
Submitting realization: "uc_phys__phys5"
        Submitting Chunk 1:     2011-08-28_12:00:00     2011-08-29_00:00:00
        Submitting Chunk 2:     2011-08-29_00:00:00     2011-08-29_12:00:00
        Submitting Chunk 3:     2011-08-29_12:00:00     2011-08-30_00:00:00

[user@mycomputer~]$ wrf4g_status --long
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
uc_phys__phys1       0     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys2       3     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys3       6     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys4       9     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys5       12    W  1/3    -          -          Submitted      - 0.00

[user@mycomputer~]$ wrf4g_status --long
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
uc_phys__phys1       0     R  1/3    mycomputer ciclogenes WRF            - 0.00
uc_phys__phys2       3     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys3       6     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys4       9     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys5       12    W  1/3    -          -          Submitted      - 0.00

As the experiment is working now, we are going to stop the ckunks of the uc_phys__phys1 realization using wrf4g_kill command.

[user@mycomputer~]$ wrf4g_kill -r uc_phys__phys1

[user@mycomputer~]$ wrf4g_status --long
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
uc_phys__phys1       -     P  0/3    -          -          Prepared       - 0.00
uc_phys__phys2       3     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys3       6     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys4       9     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys5       12    W  1/3    -          -          Submitted      - 0.00

Note that, Run.Sta has changed into Prepared value. If you want you submit again the realization, you only need to execute wrf4g_submit -r uc_phys__phys1.

[user@mycomputer~]$ wrf4g_submit -r uc_phys__phys1
Submitting realization: "uc_phys__phys1"
        Submitting Chunk 1:     2011-08-28_12:00:00     2011-08-29_00:00:00
        Submitting Chunk 2:     2011-08-29_00:00:00     2011-08-29_12:00:00
        Submitting Chunk 3:     2011-08-29_12:00:00     2011-08-30_00:00:00

[user@mycomputer~]$ wrf4g_status --long
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
uc_phys__phys1       15    W  1/3    -          -          Submitted      - 0.00
uc_phys__phys2       3     R  1/3    mycomputer ciclogenes real           - 0.00
uc_phys__phys3       6     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys4       9     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys5       12    W  1/3    -          -          Submitted      - 0.00

[user@mycomputer~]$ wrf4g_status --long
Realization          GW  Stat Chunks Comp.Res   WN         Run.Sta       ext   %
uc_phys__phys1       15    W  1/3    -          -          Submitted      - 0.00
uc_phys__phys2       3     R  1/3    mycomputer ciclogenes WRF            - 0.00
uc_phys__phys3       6     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys4       9     W  1/3    -          -          Submitted      - 0.00
uc_phys__phys5       12    W  1/3    -          -          Submitted      - 0.00

The uc_phys__phys2 is running because all realizations are independents.

Now, try to stop the uc_phys experiment and then resubmit it again