wiki:ESGFLog

Version 45 (modified by zequi, 4 years ago) (diff)

--

ESGF node at Unican (data.meteo.unican.es) log history

History of changes to ESGF node at Unican. https://github.com/ESGF/esgf-installer/wiki

Index

  1. (5 March 2018) Upgrade to v-2.5.6 (Asgard release)
  2. (5 March 2018) Republication of CORDEX datasets with fixed up dataset_root
  3. (13 March 2018) Upgrade to v-2.6.7
  4. (22 March 2018) Certificate rekey
  5. (23 March 2018) Fixed certificate chain
  6. (29 March 2018) NOAA/ESRL node decommission
  7. (28 May 2018) NFS Issue
  8. (13 Aug 2018) NFS Change

(5 March 2018) Upgrade to v-2.5.6 (Asgard release)

(5 March 2018) Republication of CORDEX datasets with fixed up dataset_root

  • Unpublish
esgunpublish --project cordex --map mapfiles/ --database-delete

If manual action is needed, psql -U psql -d esgcet, esginitialize -d VERSION and remove manually /esg/content/thredds/esgcet/.

  • Fixed dataset_root. Previously I had defined various dataset roots and now only 'esgcet' dataset root will be available.
thredds_dataset_roots =
        esg_dataroot | /esg/data
        esgcet | /data
  • Fetch-ini (with the one installed by the installer I get 'No config option: Option not found' for 'rcm_name_map')
esgprep fetch-ini --project cordex
  • Modify /esg/config/esgcet/esg.cordex.ini
variable_options = aclwdnt, alb, areacella, clfr1000, clfr200, clfr300, clfr400, clfr500, clfr600, clfr700, clfr850, clfr875, clfr900, clfr925, clfr950, clfr975, clh, clice1000, clice200, clice300, clice400, clice500, clice600, clice700, clice850, clice875, clice900, clice925, clice950, clice975, clivi, cll, clm, clt, clwmr1000, clwmr200, clwmr300, clwmr400, clwmr500, clwmr600, clwmr700, clwmr850, clwmr875, clwmr900, clwmr925, clwmr950, clwmr975, clwvi, evspsbl, evspsblpot, hfls, hfss, hufs, hur1000, hur200, hur300, hur400, hur500, hur600, hur700, hur850, hur875, hur900, hur925, hur950, hur975, hurs, hus1000, hus200, hus300, hus400, hus500, hus600, hus700, hus850, hus875, hus900, hus925, hus950, hus975, huss, mrfso, mross, mrro, mrros, mrso, mrsofc, mrsos, mrsosat, mrsosd, mrsowp, orog, pr, prc, prhmax, prls, prsn, prw, ps, psl, rlds, rlus, rlut, rsds, rsdt, rsus, rsut, sfcWind, sfcWindmax, sfcWindmaxmax, sftgif, sftlf, sic, slev, slw, snc, snd, snm, snownc, snw, sst, sund, ta1000, ta200, ta300, ta400, ta500, ta600, ta700, ta850, ta875, ta900, ta925, ta950, ta975, tas, tasmax, tasmaxts, tasmin, tasmints, tauu, tauv, ts, tsmax, tsmin, tsos, u200, u500, u850, ua1000, ua200, ua300, ua400, ua500, ua600, ua700, ua850, ua875, ua900, ua925, ua950, ua975, uas, ustar, v200, v500, v850, va1000, va200, va300, va400, va500, va600, va700, va850, va875, va900, va925, va950, va975, vas, wsgsmax, zg1000, zg200, zg300, zg350, zg400, zg450, zg500, zg550, zg600, zg650, zg700, zg750, zg800, zg850, zg875, zg900, zg925, zg950, zg975, zmla

rcm_version_options = v1, v01, v2, v3, v4, v411, v1a, v5, v02

time_frequency_options = day, fx, mon, sem, 3hr, 6hr

las_time_delta_map = map(time_frequency : las_time_delta)
    mon | 1 month
    day | 1 day
    fx  | fixed
    3hr | 3 hours
    sem | 1 season
    6hr | 6 hours

# add this to rcm_name_map
UCAN-WRF341I        | WRF341I
  • Publication
# In /data/cordex/01_EuroCORDEX_INTERIM_022
esgprep drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep mapfile --project cordex CORDEX/

# In /data/cordex/02_EuroCORDEX_INTERIM_044
esgprep drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v01
esgprep drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v01
esgprep drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v01
esgprep drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044/ --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v01
esgprep mapfile --project cordex CORDEX/

esgprep drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --rescan
esgprep drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --rescan
esgprep drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/02_EuroCORDEX_INTERIM_044_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --rescan
esgprep mapfile --project cordex CORDEX/
esgpublish --service fileservice --map mapfiles/ --project cordex --thredds --publish

# In /data/cordex/01_EuroCORDEX_INTERIM_022
esgprep drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v02
esgprep drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v02 --rescan
esgprep drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v02 --rescan
esgprep drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022_v20140616/ --version 20140616 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v02 --rescan
esgprep mapfile --project cordex CORDEX/
esgpublish --service fileservice --map mapfiles/ --project cordex --thredds --publish

# In /data/cordex/EUR-44_CCCma-CanESM2_rcp4.5_r1i1p1_UCAN-WRF341I_v2, using --rename-destination from antonio
# Add filename_dst_format = %(variable)s_%(domain)s_%(driving_model)s_%(experiment)s_%(ensemble)s_%(rcm_model)s_%(rcm_version)s_%(time_frequency)s_%(period_start)s-%(period_end)s.nc to esg.cordex.ini
# and set filename_format to filename_format = %(variable)s_%(domain)s_%(driving_model)s_%(experiment)s_%(ensemble)s_%(rcm_model)s_%(rcm_version)s_%(time_frequency)s[_%(period_start)s-%(period_end)s].nc

/root/esgf-prepare/esgprep/esgprep.py drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_CCCma-CanESM2_rcp4.5_r1i1p1_UCAN-WRF341I_v2_v20161020/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v2 --set-value experiment=rcp45 --rename-destination
/root/esgf-prepare/esgprep/esgprep.py drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_CCCma-CanESM2_rcp4.5_r1i1p1_UCAN-WRF341I_v2_v20161020/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v2 --set-value experiment=rcp45 --rename-destination
/root/esgf-prepare/esgprep/esgprep.py drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_CCCma-CanESM2_rcp4.5_r1i1p1_UCAN-WRF341I_v2_v20161020/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v2 --set-value experiment=rcp45 --rename-destination
/root/esgf-prepare/esgprep/esgprep.py drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_CCCma-CanESM2_rcp4.5_r1i1p1_UCAN-WRF341I_v2_v20161020/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex --set-value rcm_version=v2 --set-value experiment=rcp45 --rename-destination

rm -rf CORDEX/output/EUR-44/UCAN/CCCma-CanESM2/rcp45/r0i0p0/

# fixed orog and sftlf can be found in /oceano/gmeteo/users/zequi/esgf-fixed, copy them in /data/cordex/EUR-44_CCCma-CanESM2_rcp4.5_r1i1p1_UCAN-WRF341I_v2/fixed and rename files with 'rcp4.5' to 'rcp45'

# ls /data/cordex/EUR-44_CCCma-CanESM2_rcp4.5_r1i1p1_UCAN-WRF341I_v2/fixed
# orog_EUR-44_CCCma-CanESM2_rcp45_r0i0p0_UCAN-WRF341I_v2_fx.nc
# sftlf_EUR-44_CCCma-CanESM2_rcp45_r0i0p0_UCAN-WRF341I_v2_fx.nc

esgprep drs list --project cordex fixed/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs tree --project cordex fixed/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs todo --project cordex fixed/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs upgrade --project cordex fixed/ --version 20161020 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex

esgprep mapfile --project cordex CORDEX/
esgpublish --service fileservice --map mapfiles/ --project cordex --thredds --publish

# In /data/cordex/EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1
esgprep drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex

# Removed unfixed datasets for variables orog y sftlf
rm -rf CORDEX/output/EUR-44/UCAN/ECMWF-ERAINT/evaluation/r0i0p0/
mkdir fixed
# Copy from /oceano/gmeteo/users/zequi/esgf-fixed
# ls fixed/
# orog_EUR-44_ECMWF-ERAINT_evaluation_r0i0p0_UCAN-WRF341I_v1_fx.nc
# sftlf_EUR-44_ECMWF-ERAINT_evaluation_r0i0p0_UCAN-WRF341I_v1_fx.nc

esgprep drs list --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs tree --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs todo --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs upgrade --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex

esgprep mapfile --project cordex CORDEX/
esgpublish --service fileservice --map mapfiles/ --project cordex --thredds --publish

# In /data/cordex/SAM-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1
esgprep drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/SAM-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/SAM-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/SAM-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/SAM-44_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF341I_v1_v20160531/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex

rm -rf CORDEX/output/SAM-44/UCAN/ECMWF-ERAINT/evaluation/r0i0p0/
# Copy from /oceano/gmeteo/users/zequi/esgf-fixed
# ls fixed/
# orog_SAM-44_ECMWF-ERAINT_evaluation_r0i0p0_UCAN-WRF341I_v1_fx.nc
# sftlf_SAM-44_ECMWF-ERAINT_evaluation_r0i0p0_UCAN-WRF341I_v1_fx.nc

esgprep drs list --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs tree --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs todo --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex
esgprep drs upgrade --project cordex fixed/ --version 20160531 --symlink --set-value institute_id=UCAN --set-value product=output --set-value project=cordex

esgprep mapfile --project cordex CORDEX/
esgpublish --service fileservice --map mapfiles/ --project cordex --thredds --publish

(13 March 2018) Upgrade to v-2.6.7

(22 March 2018) Certificate rekey

(23 March 2018) Fixed certificate chain

  • [root@data /etc/certs] # esg-node --install-keypair hostcert.pem hostkey.pem hostcert and hostkey are Unican TERENA credentials
  • When asked for cachain, use the one from Prashanth (email on 22 March 2018, unican_chain.pem)
  • Current status of the certificates is the following:
    • For httpd, hostcert and hostkey are the ones issued by TERENA to unican and cachain is the one from Prasanth (see email on 22 March 2018, unican_chain.pem).
    • For tomcat, the keystore contains the entry my_esgf_node with the same hostcert than httpd and the same cachain.
    • For httpd and tomcat, you can use the command esg-node --install-keypair hostcert.pem hostkey.pem and when asked for the cachain, select the appropriate one.
    • For GridFTP, the certificate in use is the one issued to us by Prashanth (see email on 29 December 2017, attachment data.meteo.unican.es.tgz).

(29 March 2018) NOAA/ESRL node decommission

  • Edited /esg/config/esgf_known_providers and removed the following lines:
      <OP>
      <NAME>NOAA Enviromental System Research Laboratory (ESRL)</NAME>
      <URL>https://esgf.esrl.noaa.gov/esgf-idp/openid/</URL>
     </OP>
    
  • /esg/config/esgf_idp_static.xml not edited (I think there is no reference to NOAA/ESRL)
  • esg-node restart

(28 May 2018) NFS Issue

A disk in Seal reported that it was failing, so the NFS data on data.meteo.unican.es was inaccessible. After fixing the issue with the disk, it was impossible to stop the NFS daemon, due to zombie processes in the machine (ps aux shows a process that looks like root 11972 0.0 0.0 0 0 ? D Jul24 0:00 [192.168.202.135]). This could be cause because reboot tries to stop the service using the init file, next time try reboot -f (see https://unix.stackexchange.com/questions/64280/what-is-the-difference-between-reboot-init-6-and-shutdown-r-now/293323).

Performing a reboot in the machine should be enough to fix the problems with no further complications does not seem to work. Host responds to ping but cannot telnet to port 22.

(13 Aug 2018) NFS Change

References: https://meteo.unican.es/trac/wiki/Equipos/Seal/NFSServer/NfsEsgf

  • Added soft option into /etc/auto.nfs4
  • Disabled autofs with chkconfig autofs off and nfs mounted manually with mount 192.168.202.135:/oceano/gmeteo/DATA/ESGF/UNICAN-NODE /vols/seal/oceano/gmeteo/DATA/ESGF/UNICAN-NODE and mount --bind /vols/seal/oceano/gmeteo/DATA/ESGF/UNICAN-NODE /esg/gridftp_root/oceano/gmeteo/DATA/ESGF/UNICAN-NODE

(25 Oct 2018) Enabled autofs

chkconfig autofs on Still don't know why the NFS client halts every few days.

Attachments (3)