wiki:ESGFNodeTutorial

Version 9 (modified by zequi, 4 years ago) (diff)

--

ESGF Local Node Deployment Tutorial

This page shows how to deploy an ESGF Node that provides data, index and idp services and belongs to the esgf-test federation. The purpose of this node is to test the process of publication in the ESGF.

This page assumes that command are executed by the root user (not sudo).

Index

  1. Previous installation clean up
  2. Installation from scratch
  3. Configuration for publishing
  4. Known issues
  5. References

1. Previous installation clean up

Execute /usr/local/bin/esg-node stop in order to stop the current ESGF services (in case they are running).

[root@spock ~]# /usr/local/bin/esg-node stop


  EEEEEEEEEEEEEEEEEEEEEE   SSSSSSSSSSSSSSS         GGGGGGGGGGGGGFFFFFFFFFFFFFFFFFFFFFF
  E::::::::::::::::::::E SS:::::::::::::::S     GGG::::::::::::GF::::::::::::::::::::F
  E::::::::::::::::::::ES:::::SSSSSS::::::S   GG:::::::::::::::GF::::::::::::::::::::F
  EE::::::EEEEEEEEE::::ES:::::S     SSSSSSS  G:::::GGGGGGGG::::GFF::::::FFFFFFFFF::::F
    E:::::E       EEEEEES:::::S             G:::::G       GGGGGG  F:::::F       FFFFFF
    E:::::E             S:::::S            G:::::G                F:::::F
    E::::::EEEEEEEEEE    S::::SSSS         G:::::G                F::::::FFFFFFFFFF
    E:::::::::::::::E     SS::::::SSSSS    G:::::G    GGGGGGGGGG  F:::::::::::::::F
    E:::::::::::::::E       SSS::::::::SS  G:::::G    G::::::::G  F:::::::::::::::F
    E::::::EEEEEEEEEE          SSSSSS::::S G:::::G    GGGGG::::G  F::::::FFFFFFFFFF
    E:::::E                         S:::::SG:::::G        G::::G  F:::::F
    E:::::E       EEEEEE            S:::::S G:::::G       G::::G  F:::::F
  EE::::::EEEEEEEE:::::ESSSSSSS     S:::::S  G:::::GGGGGGGG::::GFF:::::::FF
  E::::::::::::::::::::ES::::::SSSSSS:::::S   GG:::::::::::::::GF::::::::FF
  E::::::::::::::::::::ES:::::::::::::::SS      GGG::::::GGG:::GF::::::::FF
  EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS           GGGGGG   GGGGFFFFFFFFFFF.llnl.gov

Checking that you have root privs on spock.meteo.unican.es... [OK]
Checking requisites... 

Using IP: 193.144.184.40
Stopping search services...
Using solr_workdir=/usr/local/src/esgf/workbench/esg/solr-5.5.3
Using solr_install_dir=/usr/local/solr-home/slave-8983
Using solr_data_dir=/esg/solr-index/slave-8983
Using solr_server_dir=/usr/local/solr
Using solr_logs_dir=/esg/solr-logs
Using esg_dist_url=http://esg-dn2.nsc.liu.se/esgf/dist
sudo: source: command not found
Sending stop command to Solr running on port 8983 ... waiting 5 seconds to allow Jetty process 16339 to stop gracefully.
Sending stop command to Solr running on port 8984 ... waiting 5 seconds to allow Jetty process 16554 to stop gracefully.
Stopping Globus Services for Data-Node... (GridFTP) stop_globus_services for datanode
globus-gridftp-server: unrecognized service
Stopping Globus Services for Index-Node... (MyProxy server) stop_globus_services for gateway
Stopping myproxy-server:                                   [  OK  ]
No MyProxy Process Currently Running...
Tomcat (jsvc) process is running... 

stop tomcat: /usr/local/tomcat/bin/jsvc -pidfile /var/run/tomcat-jsvc.pid -stop org.apache.catalina.startup.Bootstrap
(please wait)
postmaster (pid  16024) is running...
Stopping postgresql service:                               [  OK  ]
Stopping httpd:                                            [  OK  ]
Running shutdown hooks...

---------------------------
Running Node Services... 
node type: [ data index idp compute ] (60) 
---------------------------
---------------------------

Execute source /usr/local/bin/esg-purge.sh && esg-purge all

2. Installation from scratch

Change directory to /usr/local/bin/

[root@spock ~]# cd /usr/local/bin/

[root@spock bin]# wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/devel/esgf-installer/2.4/esg-bootstrap --no-check-certificate
[root@spock bin]# chmod 555 ./esg-bootstrap
[root@spock bin]# ./esg-bootstrap

Your directory should look like this:

[root@spock bin]# ls
esg-bootstrap  esg-functions  esg-init  esg-node  esg-purge.sh  jar_security_scan  setup-autoinstall

Check your node's version:

[root@spock bin]# ./esg-node --version
Version: v2.4.24-master-release
Release: Bifrost
Earth Systems Grid Federation (http://esgf.llnl.gov)
ESGF Node Installation Script

Set node's type:

[root@spock bin]# ./esg-node --set-type idp data index
node type set to: [ index data idp ] (28) 

Install the node:

[root@spock bin]# ./esg-node --install
Please select the ESGF distribution mirror for this installation (fastest to slowest): 
        -------------------------------------------
         [1] http://dist.ceda.ac.uk/esgf 
         [2] http://esg-dn2.nsc.liu.se/esgf 
         [3] http://aims1.llnl.gov/esgf 
         [4] http://distrib-coffee.ipsl.jussieu.fr/pub/esgf 
        -------------------------------------------
select [1] > 1
Are you ready to begin the installation? [Y/n] 
Configured host IP address does not match available IPs...
Detected multiple IP addresses bound to this host...
Please select the IP address to use for this installation
        -------------------------------------------
        [0] : 193.xxx.xxx.xxx
        [1] : 192.xxx.xxx.xxx
        -------------------------------------------
select [] > (select the one that fits your case)
Welcome to the ESGF Node installation program! :-)

What is the fully qualified domain name of this node? [spock.meteo.unican.es]: 
What is the admin password to use for this installation? (alpha-numeric only) []: 
Please re-enter password: 
What is the name of your organization? [unican]: 
Please give this node a "short" name: []: unican
Please give this node a more descriptive "long" name []: unican
What is the namespace to use for this node? (set to your reverse fqdn - Ex: "gov.llnl") [es.unican.meteo]: 
What peer group(s) will this node participate in? (esgf-test|esgf-prod) [esgf-test]: 
What is the default peer to this node? [spock.meteo.unican.es]: 
What is the hostname of the node do you plan to publish to? [spock.meteo.unican.es]: 
What email address should notifications be sent as? []: 
Is the database external to this node? [y/N]: 
Please enter the database connection string...
 (form: postgresql://[username]@[host]:[port]/esgcet)
What is the database connection string? [postgresql://dbsuper@localhost:5432/esgcet]: postgresql://
entered: postgresql://dbsuper@localhost:5432/esgcet
What is the (low priv) db account for publisher? [esgcet]: 
What is the db password for publisher user (esgcet)? []: 
Enter password for postgres user dbsuper: 
Re-enter password for postgres user dbsuper: 
Please Enter PostgreSQL port number [5432]:> 
Would you like a "system" or "user" publisher configuration: 
        -------------------------------------------
        *[1] : System
         [2] : User
        -------------------------------------------
         [C] : (Custom)
        -------------------------------------------
select [1] > 

You have selected: 1
Publisher configuration file -> [/esg/config/esgcet/esg.ini]

Is this correct? [Y/n] 
Your publisher configuration file will be: /esg/config/esgcet/esg.ini
What is your organization's id? [unican]: 
Would you like to use the DN: (OU=ESGF.ORG, O=ESGF) ? [Y/n]:
...
Please enter the password for this keystore   : 

Known issues

#error "Psycopg requires PostgreSQL client library (libpq) >= 9.1

This error occurs sometimes during installation but removing the node and installing it from scratch seems to solve it...

Traceback (most recent call last):
  File "setup.py", line 110, in <module>
    """,
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/dist.py", line 239, in __init__
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/dist.py", line 263, in fetch_build_eggs
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/pkg_resources.py", line 568, in resolve
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/pkg_resources.py", line 806, in best_match
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/pkg_resources.py", line 818, in obtain
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/dist.py", line 313, in fetch_build_egg
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 609, in easy_install
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 639, in install_item
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 825, in install_eggs
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 1031, in build_and_install
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 1019, in run_setup
distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1

Sorry...
This action did not complete successfully
Please re-run this task until successful before continuing further

Also please review the installation FAQ it may assist you
https://github.com/ESGF/esgf.github.io/wiki/ESGFNode%7CFAQ

References