wiki:NCMLUserStories

Summary

  • What is NetCDF?: network Common Data Form (NetCDF) is a self-describing, portable, scalable, appendable, sharable and archivable set of interfaces for array-oriented data access and a freely distributed collection of data access libraries for C, Fortran, C++, Java, and other languages.
  • CF-Convention: climate and forecast (CF) metadata convention has been developed for use with climate and forecast data, and comparable observational datasets, to locate data in space–time as a function of other independent variables and to identify data sufficiently to enable users of data from different sources to decide what is comparable, and to distinguish variables in archives. Data should be self-describing—no external tables needed to interpret it. Slides of Jonathan Gregory (Met Office).
  • NetCDF Markup Language (NcML): NcML is an XML representation of netCDF metadata. NcML is similar to the netCDF CDL (network Common data form Description Language), except, of course, it uses XML syntax. Schema.

Examples

  • How to modify the values of a variable in a netcdf file?

Ncml could be a useful tool to manipulate netcdf files. In this example, we are going to describe how to modify the values of a particular variable (rlat) in an existing netcdf file (tas.nc).

For this purpuse it is necessary to generate a ncml that refers to that particular netcdf, by adding a location argument. Then, we just need to specify the variable name that we want to change, their values and the separator between them. The ncml file will look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="tas.nc">
  <variable name="rlat">
    <values start="0.883846282958984" incr="0.22"/>
  </variable>
</netcdf>

In order to save the changes in a new netcdf file, we can use toolsUI.

java -cp /software/meteo/jar/toolsUI-4.6.5.jar ucar.nc2.write.Nccopy --input rlat_new.ncml --output tas_new.nc --format netcdf4_classic -d 4 -sh
  • How to define new dimensions or correct/modify existing ones?
    <?xml version="1.0" encoding="UTF-8"?>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/pp_0.25deg_reg_v12.0.nc">
         <dimension name="rlon" length="464" />
         <dimension name="rlat" length="201" />
         <dimension name="time1" length="23922" isUnlimited="false" />
    </netcdf>
    
  • How to define new variables or correct/modify existing ones?
    <?xml version="1.0" encoding="UTF-8"?>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/pp_0.25deg_reg_v12.0.nc">
         <dimension name="rlon" length="464" />
         <dimension name="rlat" length="201" />
         <dimension name="time1" length="23922" isUnlimited="false" />
         <variable name="lon" shape="rlon" type="float">
              <attribute name="standard_name" value="longitude" />
              <attribute name="long_name" value="Longitude values" />
              <attribute name="units" value="degrees_east" />
              <attribute name="axis" value="X" />
              <attribute name="_CoordinateAxisType" value="Lon" />
              <values separator=",">
                    -40.375, -40.125, -39.875, -39.625, -39.375, -39.125, -38.875, -38.625, -38.375, -38.125, -37.875, -37.625, -37.375, -37.125, -36.875, -36.625, -36.375, -36.125, -35.875, -35.625, -35.375, -35.125, -34.875, -34.625, -34.375, -34.125, -33.875, -33.625, -33.375, -33.125, -32.875, -32.625, -32.375, -32.125, -31.875, -31.625, -31.375, -31.125, -30.875, -30.625, -30.375, -30.125, -29.875, -29.625, -29.375, -29.125, -28.875, -28.625, -28.375, -28.125, -27.875, -27.625, -27.375, -27.125, -26.875, -26.625, -26.375, -26.125, -25.875, -25.625, -25.375, -25.125, -24.875, -24.625, -24.375, -24.125, -23.875, -23.625, -23.375, -23.125, -22.875, -22.625, -22.375, -22.125, -21.875, -21.625, -21.375, -21.125, -20.875, -20.625, -20.375, -20.125, -19.875, -19.625, -19.375, -19.125, -18.875, -18.625, -18.375, -18.125, -17.875, -17.625, -17.375, -17.125, -16.875, -16.625, -16.375, -16.125, -15.875, -15.625, -15.375, -15.125, -14.875, -14.625, -14.375, -14.125, -13.875, -13.625, -13.375, -13.125, -12.875, -12.625, -12.375, -12.125, -11.875, -11.625, -11.375, -11.125, -10.875, -10.625, -10.375, -10.125, -9.875, -9.625, -9.375, -9.125, -8.875, -8.625, -8.375, -8.125, -7.875, -7.625, -7.375, -7.125, -6.875, -6.625, -6.375, -6.125, -5.875, -5.625, -5.375, -5.125, -4.875, -4.625, -4.375, -4.125, -3.875, -3.625, -3.375, -3.125, -2.875, -2.625, -2.375, -2.125, -1.875, -1.625, -1.375, -1.125, -0.875, -0.625, -0.375, -0.125, 0.125, 0.375, 0.625, 0.875, 1.125, 1.375, 1.625, 1.875, 2.125, 2.375, 2.625, 2.875, 3.125, 3.375, 3.625, 3.875, 4.125, 4.375, 4.625, 4.875, 5.125, 5.375, 5.625, 5.875, 6.125, 6.375, 6.625, 6.875, 7.125, 7.375, 7.625, 7.875, 8.125, 8.375, 8.625, 8.875, 9.125, 9.375, 9.625, 9.875, 10.125, 10.375, 10.625, 10.875, 11.125, 11.375, 11.625, 11.875, 12.125, 12.375, 12.625, 12.875, 13.125, 13.375, 13.625, 13.875, 14.125, 14.375, 14.625, 14.875, 15.125, 15.375, 15.625, 15.875, 16.125, 16.375, 16.625, 16.875, 17.125, 17.375, 17.625, 17.875, 18.125, 18.375, 18.625, 18.875, 19.125, 19.375, 19.625, 19.875, 20.125, 20.375, 20.625, 20.875, 21.125, 21.375, 21.625, 21.875, 22.125, 22.375, 22.625, 22.875, 23.125, 23.375, 23.625, 23.875, 24.125, 24.375, 24.625, 24.875, 25.125, 25.375, 25.625, 25.875, 26.125, 26.375, 26.625, 26.875, 27.125, 27.375, 27.625, 27.875, 28.125, 28.375, 28.625, 28.875, 29.125, 29.375, 29.625, 29.875, 30.125, 30.375, 30.625, 30.875, 31.125, 31.375, 31.625, 31.875, 32.125, 32.375, 32.625, 32.875, 33.125, 33.375, 33.625, 33.875, 34.125, 34.375, 34.625, 34.875, 35.125, 35.375, 35.625, 35.875, 36.125, 36.375, 36.625, 36.875, 37.125, 37.375, 37.625, 37.875, 38.125, 38.375, 38.625, 38.875, 39.125, 39.375, 39.625, 39.875, 40.125, 40.375, 40.625, 40.875, 41.125, 41.375, 41.625, 41.875, 42.125, 42.375, 42.625, 42.875, 43.125, 43.375, 43.625, 43.875, 44.125, 44.375, 44.625, 44.875, 45.125, 45.375, 45.625, 45.875, 46.125, 46.375, 46.625, 46.875, 47.125, 47.375, 47.625, 47.875, 48.125, 48.375, 48.625, 48.875, 49.125, 49.375, 49.625, 49.875, 50.125, 50.375, 50.625, 50.875, 51.125, 51.375, 51.625, 51.875, 52.125, 52.375, 52.625, 52.875, 53.125, 53.375, 53.625, 53.875, 54.125, 54.375, 54.625, 54.875, 55.125, 55.375, 55.625, 55.875, 56.125, 56.375, 56.625, 56.875, 57.125, 57.375, 57.625, 57.875, 58.125, 58.375, 58.625, 58.875, 59.125, 59.375, 59.625, 59.875, 60.125, 60.375, 60.625, 60.875, 61.125, 61.375, 61.625, 61.875, 62.125, 62.375, 62.625, 62.875, 63.125, 63.375, 63.625, 63.875, 64.125, 64.375, 64.625, 64.875, 65.125, 65.375, 65.625, 65.875, 66.125, 66.375, 66.625, 66.875, 67.125, 67.375, 67.625, 67.875, 68.125, 68.375, 68.625, 68.875, 69.125, 69.375, 69.625, 69.875, 70.125, 70.375, 70.625, 70.875, 71.125, 71.375, 71.625, 71.875, 72.125, 72.375, 72.625, 72.875, 73.125, 73.375, 73.625, 73.875, 74.125, 74.375, 74.625, 74.875, 75.125, 75.375
              </values>          
         </variable>
         <variable name="lat" shape="rlat" type="float">
              <attribute name="standard_name" value="latitude" />
              <attribute name="long_name" value="Latitude values" />
              <attribute name="units" value="degrees_north" />
              <attribute name="axis" value="Y" />
              <attribute name="_CoordinateAxisType" value="Lat" />
         </variable>
         <variable name="time1" shape="time1" type="double">
              <attribute name="standard_name" value="time" />
              <attribute name="long_name" value="Time in days" />
              <attribute name="units" value="days since 1950-01-01 00:00:00" />
              <attribute name="calendar" value="standard" />
              <attribute name="_CoordinateAxisType" value="Time" />
              <values start="0" increment="1"/>
         </variable>
    </netcdf>
    
  • How to aggregate datasets?
    <?xml version="1.0" encoding="UTF-8"?>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/pp_0.25deg_reg_v12.0.nc">
         <aggregation type="union">    
              <netcdf location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/elev_0.25deg_reg_v10.0.nc"/>
              <netcdf location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/pp_0.25deg_reg_v10.0.nc"/>
              <netcdf location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/rr_0.25deg_reg_v10.0.nc"/>
              <netcdf location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/tg_0.25deg_reg_v10.0.nc"/>
              <netcdf location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/tn_0.25deg_reg_v10.0.nc"/>
              <netcdf location="dods://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/tx_0.25deg_reg_v10.0.nc">
                   <variable name="tasmax" orgName="tx"/>
              </netcdf>
         </aggregation> 
    </netcdf>
    
  • How to aggregate files on an existing dimension?
    <?xml version="1.0" encoding="UTF-8"?>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
        <attribute name="svn_id" value="$Id: ncepReanalysis1_4xDaily.ncml 654 2015-03-05 17:27:46Z sixto $"/>
        <dimension name="time" length="2924" isUnlimited="false"/>
        <variable name="time" shape="time" type="float">
            <attribute name="units" value="hours since 1948-01-01 00:00:00"/>
            <attribute name="_CoordinateAxisType" value="Time" />
            <values start="0" increment="6" />
        </variable>
            <aggregation dimName="time" type="joinExisting">
                    <netcdf location="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/surface_gauss/air.2m.gauss.1948.nc" ncoords="1464"/>
                    <netcdf location="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/surface_gauss/air.2m.gauss.1949.nc" ncoords="1460"/>
                    <variable name="tas" orgName="air">
                            <attribute name="_CoordinateAxes" value="time rlat rlon"/>
                    </variable>
                    <dimension name="rlon" orgName="lon"/>
                    <variable name="rlon" shape="rlon" type="float">
                            <attribute name="standard_name" value="longitude"/>
                            <attribute name="long_name" value="longitude"/>
                            <attribute name="_CoordinateAxisType" value="Lon"/>
                            <attribute name="units" value="degrees_east"/>
                            <values>0.0 1.875 3.75 5.625 7.5 9.375 11.25 13.125 15.0 16.875 18.75 20.625 22.5 24.375 26.25 28.125 30.0 31.875 33.75 35.625 37.5 39.375 41.25 43.125 45.0 46.875 48.75 50.625 52.5 54.375 56.25 58.125 60.0 61.875 63.75 65.625 67.5 69.375 71.25 73.125 75.0 76.875 78.75 80.625 82.5 84.375 86.25 88.125 90.0 91.875 93.75 95.625 97.5 99.375 101.25 103.125 105.0 106.875 108.75 110.625 112.5 114.375 116.25 118.125 120.0 121.875 123.75 125.625 127.5 129.375 131.25 133.125 135.0 136.875 138.75 140.625 142.5 144.375 146.25 148.125 150.0 151.875 153.75 155.625 157.5 159.375 161.25 163.125 165.0 166.875 168.75 170.625 172.5 174.375 176.25 178.125 180.0 181.875 183.75 185.625 187.5 189.375 191.25 193.125 195.0 196.875 198.75 200.625 202.5 204.375 206.25 208.125 210.0 211.875 213.75 215.625 217.5 219.375 221.25 223.125 225.0 226.875 228.75 230.625 232.5 234.375 236.25 238.125 240.0 241.875 243.75 245.625 247.5 249.375 251.25 253.125 255.0 256.875 258.75 260.625 262.5 264.375 266.25 268.125 270.0 271.875 273.75 275.625 277.5 279.375 281.25 283.125 285.0 286.875 288.75 290.625 292.5 294.375 296.25 298.125 300.0 301.875 303.75 305.625 307.5 309.375 311.25 313.125 315.0 316.875 318.75 320.625 322.5 324.375 326.25 328.125 330.0 331.875 333.75 335.625 337.5 339.375 341.25 343.125 345.0 346.875 348.75 350.625 352.5 354.375 356.25 358.125</values>
                    </variable>
                    <dimension name="rlat" orgName="lat"/>
                    <variable name="rlat" shape="rlat" type="float">
                            <attribute name="standard_name" value="latitude"/>
                            <attribute name="long_name" value="latitude"/>
                            <attribute name="_CoordinateAxisType" value="Lat"/>
                            <attribute name="units" value="degrees_north"/>
                            <values>88.542 86.6531 84.7532 82.8508 80.9473 79.0435 77.1394 75.2351 73.3307 71.4262 69.5217 67.6171 65.7125 63.8079 61.9033 59.9986 58.0939 56.1893 54.2846 52.3799 50.4752 48.5705 46.6658 44.7611 42.8564 40.9517 39.047 37.1422 35.2375 33.3328 31.4281 29.5234 27.6186 25.7139 23.8092 21.9044 19.9997 18.095 16.1902 14.2855 12.3808 10.47604 8.57131 6.66657 4.76184 2.8571 0.952368 -0.952368 -2.8571 -4.76184 -6.66657 -8.57131 -10.47604 -12.3808 -14.2855 -16.1902 -18.095 -19.9997 -21.9044 -23.8092 -25.7139 -27.6186 -29.5234 -31.4281 -33.3328 -35.2375 -37.1422 -39.047 -40.9517 -42.8564 -44.7611 -46.6658 -48.5705 -50.4752 -52.3799 -54.2846 -56.1893 -58.0939 -59.9986 -61.9033 -63.8079 -65.7125 -67.6171 -69.5217 -71.4262 -73.3307 -75.2351 -77.1394 -79.0435 -80.9473 -82.8508 -84.7532 -86.6531 -88.542</values>
                    </variable>
                    <remove name="lat" type="variable" />
                    <remove name="lon" type="variable" />
            </aggregation> 
    </netcdf>
    
  • How to aggregate files along a new dimension?
    <?xml version="1.0" encoding="UTF-8"?>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
            <attribute name="Conventions" value="CF-1.6, _Coordinates, UW-1.0"/>
            <attribute name="primary_variables" value="pr, psl, rls, rlut, tas, tasmax, tasmin"/>
            <attribute name="svn_id" value="$Id: day_GloSea5_12_members_seasonal_sfc.ncml 906 2015-12-30 07:43:37Z sixto $"/>
            <dimension name="member" length="4" isUnlimited="false"/>
            <variable name="member" type="string" shape="member">
                    <attribute name="standard_name" value="realization"/>
                    <attribute name="_CoordinateAxisType" value="Ensemble"/>
                    <attribute name="ref" value="http://www.uncertml.org/samples/realisation"/>
                    <values> 00 01 02 03 </values>
            </variable>
            <dimension name="time" length="20" isUnlimited="false"/>
            <variable name="time" shape="time" type="int">
                    <attribute name="long_name" value="Forecast time for ForecastModelRunCollection"/>
                    <attribute name="standard_name" value="time"/>
                    <attribute name="units" value="hours since 1996-01-25T12:00:00"/>
                    <attribute name="_CoordinateAxisType" value="Time"/>
                    <values> 0 24 48 72 96 120 144 168 192 216 240 264 288 312 336 360 384 408 432 456</values>
            </variable>
            <variable name="latitude" type="float">
                    <attribute name="_CoordinateAxisType" value="Lat"/>
            </variable>
            <variable name="longitude_0" type="float">
                    <attribute name="_CoordinateAxisType" value="Lon"/>
            </variable>
            <variable name="longitude" type="float">
                    <attribute name="_CoordinateAxisType" value="Lon"/>
            </variable>
            <variable name="pr" type="float" shape="member time latitude longitude">
                    <attribute name="coordinates" value="member time latitude longitude"/>
            </variable>
            <remove name="bnds" type="dimension"/>
            <remove name="forecast_reference_time" type="variable"/>
            <remove name="realization" type="variable"/>
            <remove name="time_bnds" type="variable"/>
            <remove name="forecast_period_bnds" type="variable"/>
            <remove name="leadtime" type="variable"/>
            <remove name="longitude_0" type="dimension"/>
            <remove name="longitude_0" type="variable"/>
            <aggregation dimName="member" type="joinNew">
                    <variableAgg name="pr"/>
                    <netcdf>
                    <aggregation dimName="time" type="joinExisting">
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r1i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r1i1p1_19960125-19960203.nc" ncoords="10"/>
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r1i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r1i1p1_19960204-19960213.nc" ncoords="10"/>
                    </aggregation>
                    </netcdf>
                    <netcdf>
                    <aggregation dimName="time" type="joinExisting">
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r2i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r2i1p1_19960125-19960203.nc" ncoords="10"/>
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r2i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r2i1p1_19960204-19960213.nc" ncoords="10"/>
                    </aggregation>
                    </netcdf>
                    <netcdf>
                    <aggregation dimName="time" type="joinExisting">
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r3i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r3i1p1_19960125-19960203.nc" ncoords="10"/>
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r3i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r3i1p1_19960204-19960213.nc" ncoords="10"/>
                    </aggregation>
                    </netcdf>
                    <netcdf>
                    <aggregation dimName="time" type="joinExisting">
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r4i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r4i1p1_19960125-19960203.nc" ncoords="10"/>
                            <netcdf location="//oceano/gmeteo/WORK/DATA/SPECS/output/MOHC/GloSea5/seasonal/ftp.ceda.ac.uk/badc/specs/data/SPECS/output/MOHC/GloSea5/seasonal/S19960125/day/atmos/day/pr/r4i1p1/v20150930/pr_day_GloSea5_seasonal_S19960125_r4i1p1_19960204-19960213.nc" ncoords="10"/>
                    </aggregation>
                    </netcdf>
            </aggregation>
    </netcdf>
    
Last modified 9 months ago Last modified on Feb 25, 2021 10:18:00 AM