| Version 12 (modified by zequi, 5 years ago) (diff) |
|---|
CORDEX data publication
Previous configuration
See 3. Configuration for publishing
CORDEX publication
This documentation gives instructions to publish 02_EuroCORDEX_INTERIM_044_v20140616 datasets.
The source code of esgprep must be modified in order to work with symbolic links! See https://github.com/IS-ENES-Data/esgf-prepare/pull/3
Add the dataset root to /esg/config/esgcet/esg.ini:
thredds_dataset_roots =
esg_dataroot | /esg/data
01_EuroCORDEX_INTERIM_022 | /esg/cordex/01_EuroCORDEX_INTERIM_022
02_EuroCORDEX_INTERIM_044 | /esg/cordex/02_EuroCORDEX_INTERIM_044
eur44-wrf341i-eraint | /esg/cordex/eur44-wrf341i-eraint
eur44-wrf341i-canesm | /esg/cordex/eur44-wrf341i-canesm
sam44-wrf341i-eraint | /esg/cordex/sam44-wrf341i-eraint
Modify /esg/config/esgcet/esg.cordex.ini to fit your needs. In this case:
[project:cordex]
categories =
project | enum | true | true | 0
product | enum | true | true | 1
domain | enum | true | true | 2
institute | enum | true | true | 3
driving_model | enum | false | true | 4
experiment | enum | false | true | 5
ensemble | string | false | true | 6
rcm_name | string | false | true | 7
rcm_version | enum | false | true | 8
time_frequency | enum | false | true | 9
description | text | false | false | 99
category_defaults =
project | CORDEX
filename_format = %(variable)s_%(domain)s_%(driving_model)s_%(experiment)s_%(ensemble)s_%(rcm_model)s_%(rcm_version)s_%(time_frequency)s[_%(period_start)s-%(period_end)s].nc
dataset_id = cordex.%(product)s.%(domain)s.%(institute)s.%(driving_model)s.%(experiment)s.%(ensemble)s.%(rcm_name)s.%(rcm_version)s.%(time_frequency)s.%(variable)s
directory_format = %(root)s/%(project)s/%(product)s/%(domain)s/%(institute)s/%(driving_model)s/%(experiment)s/%(ensemble)s/%(rcm_model)s/%(rcm_version)s/%(time_frequency)s/%(variable)s/%(version)s
product_options = output
domain_options = AFR-44, AFR-44i, ANT-44, ANT-44i, ARC-44, ARC-44i, AUS-44, AUS-44i, CAM-44, CAM-44i, CAS-44, CAS-44i, EAS-44, EAS-44i, EUR-11, EUR-11i, EUR-44, EUR-44i, MED-44, MED-44i, MNA-22, MNA-22i, MNA-44, MNA-44i, NAM-44, NAM-44i, SAM-44, SAM-44i, WAS-44, WAS-44i
institute_options = AUTH-LHTEE, AUTH-Met, AWI, BCCR, CCCma, CHMI, CLMcom, CNRM, CRP-GL, CUNI, DHMZ, DMI, ENEA, GERICS, HMS, ICTP, IDL, IITM, IPSL-INERIS, KNMI, MGO, MIUB, MOHC, MPI-CSC, NUIM, RMIB-UGent, SMHI, UCAN, UCLM, UHOH, ULg, UQAM
driving_model_options = CCCma-CanESM2, CSIRO-QCCCE-CSIRO-Mk3-6-0, CNRM-CERFACS-CNRM-CM5, ECMWF-ERAINT, ICHEC-EC-EARTH, IPSL-IPSL-CM5A-LR, IPSL-IPSL-CM5A-MR, MIROC-MIROC5, MOHC-HadGEM2-ES, MPI-M-MPI-ESM-LR, MPI-M-MPI-ESM-MR, NCC-NorESM1-M, NOAA-GFDL-GFDL-ESM2G, NOAA-GFDL-GFDL-ESM2M
experiment_options =
cordex | evaluation | Evaluation
cordex | historical | Historical
cordex | rcp26 | RCP2.6
cordex | rcp45 | RCP4.5
cordex | rcp60 | RCP6.0
cordex | rcp85 | RCP8.5
ensemble_pattern = r%(digit)si%(digit)sp%(digit)s
rcm_model_options = AUTH-LHTEE-WRF321B, AUTH-Met-WRF331A, AWI-HIRHAM5, BCCR-WRF331, BCCR-WRF331C, CCCma-CanRCM4, CHMI-ALADIN52, CLMcom-CCLM4-8-17, CNRM-ALADIN52, CNRM-ARPEGE51, CNRM-ARPEGE52, CNRM-ALADIN53, CRP-GL-WRF331A, CUNI-RegCM4-2, DHMZ-RegCM4-2, DMI-HIRHAM5, ENEA-RegCM4-3, GERICS-REMO2009, HMS-ALADIN52, ICTP-RegCM4-3, IDL-WRF331D, IPSL-INERIS-WRF331F, IITM-RegCM4-1, IITM-RegCM4-4, KNMI-RACMO21P, KNMI-RACMO22E, KNMI-RACMO22T, MGO-RRCM, MIUB-WRF331A, MOHC-HadGEM3-RA, MOHC-HadRM3P, MPI-CSC-REMO2009, NUIM-WRF331F, SMHI-RCA4, SMHI-RCA4-SN, SMHI-RCAO, SMHI-RCAO-SN, UCAN-WRF331G, UCAN-WRF350I, UCLM-PROMES, UHOH-WRF331H, ULg-MAR36, UQAM-CRCM5,
rcm_version_options = v1, v01, v2, v3, v4, v411, v1a, v5
time_frequency_options = day, fx, mon, sem, 3hr, 6hr
maps = las_time_delta_map, domain_description_map, rcm_name_map
domain_description_map = map(project_id, domain : domain_description)
cordex | AFR-44 | Africa
cordex | AFR-44i | Africa
cordex | ANT-44 | Antarctica
cordex | ANT-44i | Antarctica
cordex | ARC-44 | The Arctic
cordex | ARC-44i | The Arctic
cordex | AUS-44 | Australasia
cordex | AUS-44i | Australasia
cordex | CAM-44 | Central America
cordex | CAM-44i | Central America
cordex | CAS-44 | Central Asia
cordex | CAS-44i | Central Asia
cordex | EAS-44 | East Asia
cordex | EAS-44i | East Asia
cordex | EUR-11 | High-res. Europe
cordex | EUR-11i | High-res. Europe
cordex | EUR-44 | Europe
cordex | EUR-44i | Europe
cordex | MED-44 | HYMEX Mediterranean
cordex | MED-44i | HYMEX Mediterranean
cordex | MNA-22 | Middle East and North Africa
cordex | MNA-22i | Middle East and North Africa
cordex | MNA-44 | Middle East and North Africa
cordex | MNA-44i | Middle East and North Africa
cordex | NAM-44 | North America
cordex | NAM-44i | North America
cordex | SAM-44 | South America
cordex | SAM-44i | South America
cordex | WAS-44 | West Asia
cordex | WAS-44i | West Asia
rcm_name_map = map(project, rcm_model : rcm_name)
cordex | AUTH-LHTEE-WRF321B | WRF321B
cordex | AUTH-Met-WRF331A | WRF331A
cordex | AWI-HIRHAM5 | HIRHAM5
cordex | BCCR-WRF331 | WRF331
cordex | BCCR-WRF331C | WRF331C
cordex | CCCma-CanRCM4 | CanRCM4
cordex | CHMI-ALADIN52 | ALADIN52
cordex | CLMcom-CCLM4-8-17 | CCLM4-8-17
cordex | CNRM-ALADIN52 | ALADIN52
cordex | CNRM-ARPEGE51 | ARPEGE51
cordex | CNRM-ARPEGE52 | ARPEGE52
cordex | CNRM-ALADIN53 | ALADIN53
cordex | CRP-GL-WRF331A | WRF331A
cordex | CUNI-RegCM4-2 | RegCM4-2
cordex | DHMZ-RegCM4-2 | RegCM4-2
cordex | DMI-HIRHAM5 | HIRHAM5
cordex | ENEA-RegCM4-3 | RegCM4-3
cordex | GERICS-REMO2009 | REMO2009
cordex | HMS-ALADIN52 | ALADIN52
cordex | ICTP-RegCM4-3 | RegCM4-3
cordex | IDL-WRF331D | WRF331D
cordex | IPSL-INERIS-WRF331F | WRF331F
cordex | IITM-RegCM4-1 | RegCM4-1
cordex | IITM-RegCM4-4 | RegCM4-4
cordex | KNMI-RACMO21P | RACMO21P
cordex | KNMI-RACMO22E | RACMO22E
cordex | KNMI-RACMO22T | RACMO22T
cordex | MGO-RRCM | RRCM
cordex | MIUB-WRF331A | WRF331A
cordex | MOHC-HadGEM3-RA | HadGEM3-RA
cordex | MOHC-HadRM3P | HadRM3P
cordex | MPI-CSC-REMO2009 | REMO2009
cordex | NUIM-WRF331F | WRF331F
cordex | RMIB-UGent-ALARO-0 | ALARO-0
cordex | SMHI-RCA4 | RCA4
cordex | SMHI-RCA4-SN | RCA4-SN
cordex | SMHI-RCAO | RCAO
cordex | SMHI-RCAO-SN | RCAO-SN
cordex | UCAN-WRF331G | WRF331G
cordex | UCAN-WRF350I | WRF350I
cordex | UCLM-PROMES | PROMES
cordex | UHOH-WRF331H | WRF331H
cordex | ULg-MAR36 | MAR36
cordex | UQAM-CRCM5 | CRCM5
las_time_delta_map = map(time_frequency : las_time_delta)
mon | 1 month
day | 1 day
fx | fixed
3hr | 3 hours
sem | seasonal mean
6hr | 6 hours
project_handler_name = basic_builtin
las_configure = true
thredds_exclude_variables = a, a_bnds, alev1, alevel, alevhalf, alt40, b, b_bnds, basin, bnds, bounds_lat, bounds_lon, dbze, depth, depth0m, depth100m, depth_bnds, geo_region, height, height10m, height2m, Lambert_Conformal, lat, lat_bnds, lat_bounds, latitude, latitude_bnds, lat_vertices, layer, lev, lev_bnds, location, lon, lon_bnds, lon_bounds, lon_vertices, longitude, longitude_bnds, olayer100m, olevel, oline, p0, p220, p500, p560, p700, p840, plev, plev3, plev7, plev8, plev_bnds, plevs, pressure1, region, rho, rlat, rotated_latitude_longitude, rotated_pole, rlon, scatratio, sdepth, sdepth1, sza5, tau, tau_bnds, time, time1, time2, time_bnds, vegtype, x, y
variable_locate = ps,ps_
variable_per_file = true
Generate the DRS structure. Use --version to change version. User --symlink to create symlinks. If some facet is incorrect or missing use --set-value (institute_id and product in this case):
esgprep drs list --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022 --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output esgprep drs tree --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022 --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output esgprep drs todo --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022 --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output esgprep drs upgrade --project cordex /oceano/gmeteo/DATA/ESGF/UNICAN-NODE/DATASETS/01_EuroCORDEX_INTERIM_022 --version 20140529 --symlink --set-value institute_id=UCAN --set-value product=output
Generate the mapfiles:
esgprep mapfile --project cordex ./CORDEX/
Publish to data and index nodes:
esgpublish --service fileservice --map mapfiles/ --project cordex --thredds --publish
Data unpublication
Unpublication is well documented in esg-publisher docs. You can use bash commands to help with unpublication:
find /a/directory/with/mapfiles -regextype posix-extended -regex '.*ustar.*\.map' -exec esgunpublish --project cordex --map {} \;
The previous command will unpublish datasets for the ustar variable, using mapfiles found in /a/directory/with/mapfiles.
