Changes between Version 7 and Version 8 of udg/ecoms/dataserver/interfaces


Ignore:
Timestamp:
Oct 27, 2015 7:21:59 PM (6 years ago)
Author:
gutierjm
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • udg/ecoms/dataserver/interfaces

    v7 v8  
    1 As an alternative to the [wiki:RPackage R package], som utilities have been developed in other environments (Matlab and Python) to remotely access the UDG datasets. In the following subsections we introduce the versions of the `loadSystem4` function developed in the Python and Matlab languages with some worked examples.
     1Although ECOMS-UDG provides a [../web web interface] to explore and access the datasets, it is strongly recommented the use of `OPeNDAP` (a.k.a. `DODS`) client libraries to remotely access the data from scientific computing environments (R, Matlab, Python, etc.). In order to facilitate this task a number of utilities for (remote) data access have been developed as part of the ECOMS-UDG.
     2
     3The [../../RPackage ecomsUDG.Raccess] '''R package''' is the main of these tools, and it is based on the [http://www.unidata.ucar.edu/software/netcdf-java/documentation.htm NetCDF Java] OPeNDAP client, using the `rJava` R package (a similar approach is followed in the '''Matlab''' interface). Alternatively, the most recent NetCDF library versions provide access to `OPeNDAP` datasets (this is the solution for the '''Python''' implementation).
    24
    35'''IMPORTANT NOTE''': Unlike the [wiki:RPackage R package], which will be continuously updated as part of the data management activities, the Python and Matlab functions are not guaranteed to be maintained and updated with the same regularity, and therefore the use of the R environment is encouraged in the framework of ECOMS.  In particular, the different datasets are not homogenized in the Matlab and Python versions and the only variables available are those originally available in the corresponding datasets (see the list of [wiki:dataserver/listofvariables available variables] for details).
     
    57  * [wiki:udg/ecoms/dataserver/interfaces/python Python Interface]
    68  * [wiki:udg/ecoms/dataserver/interfaces/matlab MatLab Interface]
     9
     10
     11
     12
     13
     14
     15{{{#!comment
     16
     17Although the `TDS` provides a web interface to explore and access the datasets (shown in [#s.web.access web access section]), it is strongly recommented the use of `OPeNDAP` (a.k.a. `DODS`) client libraries to remotely access the data from scientific computing environments (`R`, `Matlab`, `Python`, etc.). For instance, the `R` function provided in this tutorial is based on the ''NetCDF Java'' OPeNDAP client[[FootNote(http://www.unidata.ucar.edu/software/netcdf-java/documentation.htm)]], using the `rJava` `R` package (a similar approach is been also made for the `Matlab` implementation). Alternatively, the most recent ''NetCDF library'' versions provide access to `OPeNDAP` datasets (this is the solution for the `Python` implementation). In the following, we show a simple example of data access using the `R` package developed as part of the data portal. In particular the ''System4'' datasets can by directly accessed using the `loadSystem4` function, allowing the retrieval of slices for a particular variable in any of the dataset dimensions (`member`/space/`runtime`/`time`). Note that a more ellaborated worked example using `R` is shown in the [#Appendix.rexample R example section].  Moreover, for a better understanding of the datasets structure, the use of the web interface for the `OPeNDAP`  service is also illustrated [#s.web.access web access section].
     18
     19
     20= Accessing to the Data portal using Python (Pydap version) = #ex.pydap
     21
     22[[NoteBox(warn,This section **still** needs revision)]]
     23
     24{{{#!csh
     25[user@host ~]$ pip install Pydap
     26........................................................................
     27[user@host ~]$ python
     28Python 2.7.2 (default, Mar  3 2012, 10:45:44)
     29[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
     30Type "help", "copyright", "credits" or "license" for more information.
     31>>>
     32}}}
     33
     34The new registration process impose a more elaborated `http` requests. The !PyDap package is able to follow redirects but cookies are ignored. The UDG python package has a been released to help users handle the new authentication service. [raw-attachment:udg.py:wiki:EcomsUdg Download the UDG Python package]
     35
     36{{{#!python
     37>>> from udg import install_udg_client
     38>>> install_udg_client()
     39>>> from pydap.client import open_url
     40>>> udg_dataset='system4/System4_Seasonal_15Members.ncml'
     41>>> url= 'http://%s:%s@meteo.unican.es/tds5/dodsC/%s' % ('YOUR_USERNAME','YOUR_PASSWORD',udg_dataset)
     42>>> dataset = open_url(url)
     43>>> print type(dataset)
     44<class 'pydap.model.DatasetType'>
     45>>> print dataset.keys()
     46['lat', 'lon', 'run', 'time', 'time1', 'time2', 'member', 'Maximum_temperature_at_2_metres_since_last_24_hours_surface', 'Minimum_temperature_at_2_metres_since_last_24_hours_surface', 'Mean_temperature_at_2_metres_since_last_24_hours_surface', 'Total_precipitation_surface', 'Mean_sea_level_pressure_surface']
     47>>> MN2T24 = dataset['Minimum_temperature_at_2_metres_since_last_24_hours_surface']
     48>>> print MN2T24.dimensions
     49('member', 'run', 'time', 'lat', 'lon')
     50>>> print MN2T24.shape
     51(15, 360, 215, 241, 480)
     52>>> arr = MN2T24[0,11:360:12,31:62,66,475]
     53>>> print numpy.squeeze(numpy.mean(arr,2))
     54[ 270.79171753  273.29437256  271.56661987  271.03707886  271.82745361
     55  272.49279785  271.48086548  268.59121704  271.53125     273.82156372
     56  270.99401855  274.23626709  270.99328613  271.56115723  273.98986816
     57  270.50756836  272.45046997  270.65560913  271.31182861  272.77200317
     58  273.4359436   271.85021973  273.39648438  274.16384888  269.98248291
     59  271.30166626  273.11950684  271.27301025  272.29147339  270.46688843]
     60}}}
     61
     62
     63= Accessing to the Data portal using Octave = #ex.octave
     64
     65[[NoteBox(warn,This section needs revision and integrtion with the [#ex.matlab Matlab example section])]]
     66
     67{{{#!text/matlab
     68>> ver
     69----------------------------------------------------------------------
     70GNU Octave Version 3.6.1
     71GNU Octave License: GNU General Public License
     72Operating System: unknown
     73----------------------------------------------------------------------
     74}}}
     75
     76{{{#!text/matlab
     77>> urlwrite('http://www.meteo.unican.es/work/netcdfAll-4.3.jar','netcdfAll-4.3.jar')
     78>> javaaddpath('./netcdfAll-4.3.jar');
     79>> javaMethod('setGlobalCredentialsProvider','ucar.nc2.util.net.HTTPSession',javaObject('ucar.nc2.util.net.HTTPBasicProvider','username','password'));
     80>> ncfile = javaMethod('openDataset','ucar.nc2.dataset.NetcdfDataset','http://www.meteo.unican.es/tds5/dodsC/system4/System4_Seasonal_15Members.ncml');
     81>> v = ncfile.findVariable('Minimum_temperature_at_2_metres_since_last_24_hours_surface');
     82>> disp(v.getDimensions.toString)
     83[   member = 15;,    run = 360;,    time = 215;,    lat = 241;,    lon = 480;]
     84>> d = v.read('0,11:359:12,31:61,66,475');
     85>> tmp = javaObject('org.octave.Matrix',d.reduce.copyToNDJavaArray);
     86>> oldFlag = java_convert_matrix (1);
     87>> octaveMatrix = tmp.ident(tmp);
     88[ (30 by 31) array of double ]
     89>> disp(squeeze(mean(octaveMatrix,2))')
     90 Columns 1 through 13:
     91
     92   270.79   273.29   271.57   271.04   271.83   272.49   271.48   268.59   271.53   273.82   270.99   274.24   270.99
     93
     94 Columns 14 through 26:
     95
     96   271.56   273.99   270.51   272.45   270.66   271.31   272.77   273.44   271.85   273.40   274.16   269.98   271.30
     97
     98 Columns 27 through 30:
     99
     100   273.12   271.27   272.29   270.47
     101}}}
     102
     103= Accessing to the Data portal using Matlab= #ex.matlab
     104
     105[[NoteBox(warn,This section needs revision])]]
     106
     107{{{#!text/matlab
     108>> ver
     109-------------------------------------------------------------------------------------
     110MATLAB Version 7.8.0.347 (R2009a)
     111MATLAB License Number: 161051
     112Operating System: Microsoft Windows Vista Version 6.1 (Build 7601: Service Pack 1)
     113Java VM Version: Java 1.6.0_04-b12 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
     114-------------------------------------------------------------------------------------
     115>> javaaddpath('http://www.meteo.unican.es/work/netcdfAll-4.3.jar');
     116>> %javaaddpath('ftp://ftp.unidata.ucar.edu/pub/netcdf-java/v4.3/netcdfAll-4.3.jar');
     117}}}
     118
     119{{{#!text/matlab
     120>> import ucar.nc2.util.net.* %this will download the netcdfAll-4.3.jar
     121>> HTTPSession.setGlobalCredentialsProvider(HTTPBasicProvider('username','password'));
     122>> import ucar.nc2.*;
     123>> import ucar.nc2.dataset.*;
     124>> ncfile = NetcdfDataset.openDataset('http://www.meteo.unican.es/tds5/dodsC/system4/System4_Seasonal_15Members.ncml');
     125>> v = ncfile.findVariable('Minimum_temperature_at_2_metres_since_last_24_hours_surface');
     126>> disp(v.getDimensions)
     127[   member = 15;,    run = 360;,    time = 215;,    lat = 241;,    lon = 480;]
     128>> data = v.read('0,11:359:12,31:61,66,475').copyToNDJavaArray();
     129>> disp(squeeze(mean(data,3)))
     130  Columns 1 through 13
     131
     132  270.7917  273.2944  271.5666  271.0371  271.8275  272.4928  271.4809  268.5912  271.5313  273.8216  270.9940  274.2363  270.9933
     133
     134  Columns 14 through 26
     135
     136  271.5612  273.9899  270.5076  272.4505  270.6556  271.3118  272.7720  273.4359  271.8502  273.3965  274.1638  269.9825  271.3017
     137
     138  Columns 27 through 30
     139
     140  273.1195  271.2730  272.2915  270.4669
     141}}}
     142
     143[[FootNote]]
     144
     145}}}