Changes between Version 43 and Version 44 of ESGFGetCredentials


Ignore:
Timestamp:
Jan 8, 2015 12:46:26 AM (7 years ago)
Author:
antonio
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ESGFGetCredentials

    v43 v44  
    105105
    106106----
    107 = Specific use cases =
    108 
     107= Use cases =
     108
     109Some enviroment varibles can be set:
     1101. The path where the user's credentials and ESGF peers certificates will be retrived
     111{{{
     112ESGF_HOME=.esg
     113}}}
     1141. The user's !OpenId and password
     115{{{
     116OPENID=https://esgf-data.dkrz.de/esgf-idp/openid/testuser
     117OPENID_PASS=userpassword
     118}}}
     119
     120For convenience the user's credentials and trust certificates will be retrived in JKS and PEM formats:
     121{{{
     122java -jar getESGFCredentials.jar --openid $OPENID --password $OPENID_PASS --writeall --output $ESGF_HOME
     123}}}
     124
     125In the following use cases this URLs will be used:
     1261. A URL for HTTP file dowloading
     127{{{
     128HTTP_URL=http://wdcc-esgf.dkrz.de:8080/ESGF/fileServer/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ta/ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc
     129}}}
     1302. A URL for DODS/OPeNDAP access
     131{{{
     132DODS_URL=http://esgf-data1.ceda.ac.uk/thredds/dodsC/esg_dataroot/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ua/ua_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2066010103-2075123121.nc
     133}}}
    109134== Aria2 ==
    110135
     136[http://aria2.sourceforge.net aria2] is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.
     137=== File download ===
     138
     139{{{
     140aria2c --private-key=$ESGF_HOME/credentials.pem --certificate=$ESGF_HOME/credentials.pem --check-certificate=true --ca-certificate=$ESGF_HOME/ca-certificates.pem $HTTP_URL
     141}}}
     142
     143=== Retriving files from Metalink ===
    1111441. Get a metalink of ESGF Files
    112145 * Download this metalink file -> [raw-attachment:ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1.metalink example_metalink][[BR]]
     
    125158== cURL ==
    126159
    127 1. Retrieve ESGF credentials in current directory
    128 {{{
    129 java -jar getESGFCredentials-0.1.jar --openid <your_openid> --password <your_password> --credentials --cacertspem --output .
    130 }}}
    131 
    132 2. Use curl to download the file[[BR]]
    133    example_file: http://wdcc-esgf.dkrz.de:8080/ESGF/fileServer/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ta/ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc
    134 {{{
    135 curl -L -C - --cookie-jar curl-cookie --cookie curl-cookie --cert credentials.pem --cacert ca-certificates.pem -O http://wdcc-esgf.dkrz.de:8080/ESGF/fileServer/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ta/ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc
     160=== File download ===
     161{{{
     162curl --location --continue-at - --cookie curl-cookie --cert $ESGF_HOME/credentials.pem --cacert $ESGF_HOME/ca-certificates.pem -O $HTTP_URL
    136163}}}
    137164
     
    145172     * '''-O''' (`-O/--remote-name`) to write output to a local file named like the remote file we get. You can use (`-o/--output` <file-name> option) to specify the name of the file.
    146173   
    147 == WGET (OPENSSL) ==
    148 1. Retrieve ESGF credentials in $HOME/.esg
    149 {{{
    150 java -jar getESGFCredentials-0.1.jar --openid <openid> --password <password> --credentials --cacerts
    151 }}}
    152 
    153 2. Wget file[[BR]]
    154    example_file: http://wdcc-esgf.dkrz.de:8080/ESGF/fileServer/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ta/ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc
    155 {{{
    156 wget -c --ca-directory=$HOME/.esg/certificates --certificate=$HOME/.esg/credentials.pem --private-key=$HOME/.esg/credentials.pem --save-cookies=$HOME/.esg/cookies --load-cookies=$HOME/.esg/cookies ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc http://wdcc-esgf.dkrz.de:8080/ESGF/fileServer/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ta/ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc
    157 }}}
    158 
    159 
    160 == WGET (GNU TLS) ==
    161 
    162 1. Retrieve ESGF credentials in $HOME/.esg
    163 {{{
    164 java -jar getESGFCredentials-0.1.jar --openid <openid> --password <password> --credentials --cacertspem
    165 }}}
    166 
    167 2. Wget file[[BR]]
    168    example_file: http://wdcc-esgf.dkrz.de:8080/ESGF/fileServer/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ta/ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc
    169 {{{
    170 wget -c --certificate=$HOME/.esg/credentials.pem --private-key=$HOME/.esg/credentials.pem --save-cookies=$HOME/.esg/cookies --load-cookies=$HOME/.esg/cookies --ca-certificate=$HOME/.esg/ca-certificates.pem ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc http://wdcc-esgf.dkrz.de:8080/ESGF/fileServer/cmip5/output1/IPSL/IPSL-CM5A-LR/esmrcp85/6hr/atmos/6hrPlev/r1i1p1/v20120114/ta/ta_6hrPlev_IPSL-CM5A-LR_esmrcp85_r1i1p1_2016010103-2025123121.nc
    171 }}}
    172 
    173 == ToolsUI ==
    174 
    175 1. Retrieve ESGF credentials in $HOME/.esg
    176 {{{
    177 java -jar getESGFCredentials-0.1.jar --openid <openid> --password <password> --keystorejks --cacertsjks
    178 }}}
    179 
    180 2. Start ToolsUI
    181 {{{
    182 java -Dkeystore=$HOME/keystore_jks.ks -Dkeystorepassword=changeit -Dtruststore=$HOME/.esg/esg-truststore.ts -Dtruststorepassword=changeit -jar toolsUI-4.3.jar
    183 }}}
    184 
    185 3. This link is a ESGF OPENDAP resource http://data.meteo.unican.es/thredds/dodsC/esg_cordexnoncommercial/cordex/output/EUR-22/UCAN/ECMWF-ERAINT/evaluation/r1i1p1/UCAN-WRF331G/v02/3hr/clt/v20140616/clt_EUR-22_ECMWF-ERAINT_evaluation_r1i1p1_UCAN-WRF331G_v02_3hr_19790101-19791231.nc
    186 
    187 For more: http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/reference/Session.html
     174== GNU Wget ==
     175
     176== NetCDF-C ==
     177
     178The NetCDF-C libray from version 4.1 can be compiled with DAP support. Check with `nc-config` command if your NetCDF libray has been compiled with DAP support. See [https://www.unidata.ucar.edu/software/netcdf/docs/netcdf/DAP-Support.html].
     179
     180DAP access is based on [http://curl.haxx.se/libcurl/ libcurl] library. The configuration paramenters are based on a file named `.dodsrc` existing in the current working directory or user's home
     181
     182{{{
     183echo -e ' HTTP.SSL.VALIDATE=1 \n HTTP.SSL.CAPATH=$ESGF_HOME/certificates \n HTTP.SSL.CERTIFICATE=$ESGF_HOME/credentials.pem' > .dodsrc && ncdump -h $DODS_URL
     184}}}
     185
     186== NetCDF-Java ==
     187
     188The NetCDF-Java library can use the credentials and trust store by defining JVM properties as command line arguments:
     189{{{
     190NCJ_PROP=-Dkeystore=$ESGF_HOME/keystore_jks.ks -Dkeystorepassword=changeit -Dtruststore=$ESGF_HOME/esg-truststore.ts -Dtruststorepassword=changeit
     191}}}
     192
     193For more info visit [http://www.unidata.ucar.edu/software/thredds/current/netcdf-java]
     194
     195=== NCdumpW ===
     196
     197Dump DODS/OPeNDAP URL metadata:
     198
     199{{{
     200java $NCJ_PROP -cp netcdf-java/toolsUI-4.3.jar ucar.nc2.NCdumpW $DODS_URL -cdl
     201}}}
     202
     203=== ToolsUI ===
     204
     205Open a dataset (i.e. DODS/OPeNDAP) with NetCDF-Java's ToolsUI Java application:
     206{{{
     207java $NCJ -jar netcdf-java/toolsUI-4.3.jar $DODS_URL
     208}}}
    188209
    189210== ESGF WGET Script (Linux) ==
     
    197218}}}
    198219
    199 == NETCDF DAP ==
    200 https://www.unidata.ucar.edu/software/netcdf/docs/netcdf/DAP-Support.html
    201 {{{
    202 java -jar getESGFCredentials-0.0.2.jar -o <openid> -p <password> --credentials --cacerts --cacertspem --output <path>
    203 }}}
    204220= Developers Guide =
    205221== Github ==