= ESGF Local Node Deployment Tutorial = This page shows how to deploy an ESGF Node that provides data, index and idp services and belongs to the esgf-test federation. The purpose of this node is to test the process of publication in the ESGF. This page assumes that command are executed by the root user (not sudo). == Index == 1. Previous installation clean up 2. Installation from scratch 3. Configuration for publishing 4. Known issues 5. References == 1. Previous installation clean up == Execute {{{/usr/local/bin/esg-node stop}}} in order to stop the current ESGF services (in case they are running). {{{ [root@spock ~]# /usr/local/bin/esg-node stop EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGGGGGGGGGFFFFFFFFFFFFFFFFFFFFFF E::::::::::::::::::::E SS:::::::::::::::S GGG::::::::::::GF::::::::::::::::::::F E::::::::::::::::::::ES:::::SSSSSS::::::S GG:::::::::::::::GF::::::::::::::::::::F EE::::::EEEEEEEEE::::ES:::::S SSSSSSS G:::::GGGGGGGG::::GFF::::::FFFFFFFFF::::F E:::::E EEEEEES:::::S G:::::G GGGGGG F:::::F FFFFFF E:::::E S:::::S G:::::G F:::::F E::::::EEEEEEEEEE S::::SSSS G:::::G F::::::FFFFFFFFFF E:::::::::::::::E SS::::::SSSSS G:::::G GGGGGGGGGG F:::::::::::::::F E:::::::::::::::E SSS::::::::SS G:::::G G::::::::G F:::::::::::::::F E::::::EEEEEEEEEE SSSSSS::::S G:::::G GGGGG::::G F::::::FFFFFFFFFF E:::::E S:::::SG:::::G G::::G F:::::F E:::::E EEEEEE S:::::S G:::::G G::::G F:::::F EE::::::EEEEEEEE:::::ESSSSSSS S:::::S G:::::GGGGGGGG::::GFF:::::::FF E::::::::::::::::::::ES::::::SSSSSS:::::S GG:::::::::::::::GF::::::::FF E::::::::::::::::::::ES:::::::::::::::SS GGG::::::GGG:::GF::::::::FF EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGG GGGGFFFFFFFFFFF.llnl.gov Checking that you have root privs on spock.meteo.unican.es... [OK] Checking requisites... Using IP: 193.144.184.40 Stopping search services... Using solr_workdir=/usr/local/src/esgf/workbench/esg/solr-5.5.3 Using solr_install_dir=/usr/local/solr-home/slave-8983 Using solr_data_dir=/esg/solr-index/slave-8983 Using solr_server_dir=/usr/local/solr Using solr_logs_dir=/esg/solr-logs Using esg_dist_url=http://esg-dn2.nsc.liu.se/esgf/dist sudo: source: command not found Sending stop command to Solr running on port 8983 ... waiting 5 seconds to allow Jetty process 16339 to stop gracefully. Sending stop command to Solr running on port 8984 ... waiting 5 seconds to allow Jetty process 16554 to stop gracefully. Stopping Globus Services for Data-Node... (GridFTP) stop_globus_services for datanode globus-gridftp-server: unrecognized service Stopping Globus Services for Index-Node... (MyProxy server) stop_globus_services for gateway Stopping myproxy-server: [ OK ] No MyProxy Process Currently Running... Tomcat (jsvc) process is running... stop tomcat: /usr/local/tomcat/bin/jsvc -pidfile /var/run/tomcat-jsvc.pid -stop org.apache.catalina.startup.Bootstrap (please wait) postmaster (pid 16024) is running... Stopping postgresql service: [ OK ] Stopping httpd: [ OK ] Running shutdown hooks... --------------------------- Running Node Services... node type: [ data index idp compute ] (60) --------------------------- --------------------------- }}} Execute {{{source /usr/local/bin/esg-purge.sh && esg-purge all}}} == 2. Installation from scratch == Change directory to /usr/local/bin/ {{{[root@spock ~]# cd /usr/local/bin/}}} {{{ [root@spock bin]# wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/devel/esgf-installer/2.4/esg-bootstrap --no-check-certificate [root@spock bin]# chmod 555 ./esg-bootstrap [root@spock bin]# ./esg-bootstrap }}} Your directory should look like this: {{{ [root@spock bin]# ls esg-bootstrap esg-functions esg-init esg-node esg-purge.sh jar_security_scan setup-autoinstall }}} Check your node's version: {{{ [root@spock bin]# ./esg-node --version Version: v2.4.24-master-release Release: Bifrost Earth Systems Grid Federation (http://esgf.llnl.gov) ESGF Node Installation Script }}} Set node's type: {{{ [root@spock bin]# ./esg-node --set-type data idp index node type set to: [ index data idp ] (28) }}} Install the node: {{{ [root@spock bin]# ./esg-node --install }}} {{{ Please select the ESGF distribution mirror for this installation (fastest to slowest): ------------------------------------------- [1] http://dist.ceda.ac.uk/esgf [2] http://esg-dn2.nsc.liu.se/esgf [3] http://aims1.llnl.gov/esgf [4] http://distrib-coffee.ipsl.jussieu.fr/pub/esgf ------------------------------------------- select [1] > 1 }}} {{{ Are you ready to begin the installation? [Y/n] Configured host IP address does not match available IPs... Detected multiple IP addresses bound to this host... Please select the IP address to use for this installation ------------------------------------------- [0] : 193.xxx.xxx.xxx [1] : 192.xxx.xxx.xxx ------------------------------------------- select [] > (select the one that fits your case) }}} {{{ Welcome to the ESGF Node installation program! :-) What is the fully qualified domain name of this node? [spock.meteo.unican.es]: What is the admin password to use for this installation? (alpha-numeric only) []: Please re-enter password: What is the name of your organization? [unican]: Please give this node a "short" name: []: unican Please give this node a more descriptive "long" name []: unican What is the namespace to use for this node? (set to your reverse fqdn - Ex: "gov.llnl") [es.unican.meteo]: What peer group(s) will this node participate in? (esgf-test|esgf-prod) [esgf-test]: What is the default peer to this node? [spock.meteo.unican.es]: What is the hostname of the node do you plan to publish to? [spock.meteo.unican.es]: What email address should notifications be sent as? []: Is the database external to this node? [y/N]: Please enter the database connection string... (form: postgresql://[username]@[host]:[port]/esgcet) What is the database connection string? [postgresql://dbsuper@localhost:5432/esgcet]: postgresql:// entered: postgresql://dbsuper@localhost:5432/esgcet What is the (low priv) db account for publisher? [esgcet]: What is the db password for publisher user (esgcet)? []: }}} {{{ Enter password for postgres user dbsuper: Re-enter password for postgres user dbsuper: Please Enter PostgreSQL port number [5432]:> }}} {{{ Would you like a "system" or "user" publisher configuration: ------------------------------------------- *[1] : System [2] : User ------------------------------------------- [C] : (Custom) ------------------------------------------- select [1] > You have selected: 1 Publisher configuration file -> [/esg/config/esgcet/esg.ini] Is this correct? [Y/n] Your publisher configuration file will be: /esg/config/esgcet/esg.ini What is your organization's id? [unican]: }}} {{{ Would you like to use the DN: (OU=ESGF.ORG, O=ESGF) ? [Y/n]: ... Please enter the password for this keystore : }}} {{{ Enter a single ip address which would be cleared to access admin restricted pages. You will be prompted if you want to enter more ip-addresses Do you wish to allow further ips? y/n n }}} {{{ Create user credentials Please enter username for tomcat [dnode_user]: Please enter password for user, "dnode_user" [********]: Would you like to add another user? [y/N]: }}} {{{ Please Enter the public (i.e. routable) IP address of this host [193.xxx.xxx.xxx]:> Do you wish to use an external IDP peer?(N/y): }}} {{{ Do you want to continue with the Globus installation and setup? [Y/n] : Do you want to register the MyProxy server with Globus? [Y/n]: Please provide a Globus username []: YOUR-GLOBUS-USER Globus password []: }}} When finished, you should see something like this: {{{ Running Node Services... node type: [ data index idp ] (29) --------------------------- myproxy-s 23071 root 5u IPv4 1526752 0t0 TCP *:7512 (LISTEN) java 26088 solr 28u IPv6 1591850 0t0 TCP 127.0.0.1:7983 (LISTEN) java 26088 solr 92u IPv6 1591986 0t0 TCP *:8983 (LISTEN) java 26257 solr 28u IPv6 1592730 0t0 TCP 127.0.0.1:7984 (LISTEN) java 26257 solr 92u IPv6 1593098 0t0 TCP *:8984 (LISTEN) postmaste 29509 postgres 3u IPv6 1449862 0t0 TCP [::1]:5432 (LISTEN) postmaste 29509 postgres 4u IPv4 1449863 0t0 TCP 127.0.0.1:5432 (LISTEN) httpd 12706 root 4u IPv6 1512235 0t0 TCP *:80 (LISTEN) --------------------------- Finished!... In order to see if this node has been installed properly you may direct your browser to: http://spock.meteo.unican.es/thredds http://spock.meteo.unican.es/esg-orp http://spock.meteo.unican.es/ Your peer group membership -- : [esgf-test] Your specified "default" peer : [spock.meteo.unican.es] Your specified "index" peer - : [spock.meteo.unican.es] (url = http://spock.meteo.unican.es/) Your specified "idp" peer --- : [spock.meteo.unican.es] (name = SPOCK.METEO.UNICAN.ES) Your temporary certificates have been placed in /etc/tempcerts You can install them by executing this : esg-node --install-keypair /etc/tempcerts/hostcert.pem /etc/tempcerts/hostkey.pem When promped for the chainfile, specify: /etc/tempcerts/cacert.pem [Note: Use UNIX group permissions on /esg/content/thredds/esgcet to enable users to be able to publish thredds catalogs from data therein] %> chgrp -R /esg/content/thredds ------------------------------------------------------- Administrators of this node should subscribe to the esgf-node-admins@lists.llnl.gov by sending email to: majordomo@lists.llnl.gov with the body: subscribe esgf-node-admins ------------------------------------------------------- v2.4.24-master-release Writing additional settings to db. If these settings already exist, psql will report an error, but ok to disregard. ERROR: insert or update on table "permission" violates foreign key constraint "permission_user_id_fkey" DETAIL: Key (user_id)=(1) is not present in table "user". Node installation is complete. }}} Execute the following: {{{ [root@spock bin]# ./esg-node --install-keypair /etc/tempcerts/hostcert.pem /etc/tempcerts/hostkey.pem ... Please set the password for this keystore : Please re-enter the password for this keystore: ... certfile> /etc/tempcerts/cacert.pem certfile> ... Is the above information correct? [Y/n] Is the above information correct? [Y/n] }}} Restart the node: {{{ [root@spock bin]# ./esg-node restart }}} Check that everything works (https://github.com/ESGF/esgf-installer/wiki/ESGF-Post-Installation-Tests). If the CoG portal does not work, follow the instructions on https://www.earthsystemcog.org/projects/cog/install_or_upgrade. == Known issues == === #error "Psycopg requires PostgreSQL client library (libpq) >= 9.1 === This error occurs sometimes during installation but removing the node and installing it from scratch seems to solve it... {{{ Traceback (most recent call last): File "setup.py", line 110, in """, File "/usr/local/uvcdat/2.2.0/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/dist.py", line 239, in __init__ File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/dist.py", line 263, in fetch_build_eggs File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/pkg_resources.py", line 568, in resolve File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/pkg_resources.py", line 806, in best_match File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/pkg_resources.py", line 818, in obtain File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/dist.py", line 313, in fetch_build_egg File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 609, in easy_install File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 639, in install_item File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 825, in install_eggs File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 1031, in build_and_install File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg/setuptools/command/easy_install.py", line 1019, in run_setup distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1 Sorry... This action did not complete successfully Please re-run this task until successful before continuing further Also please review the installation FAQ it may assist you https://github.com/ESGF/esgf.github.io/wiki/ESGFNode%7CFAQ }}} === Failed building wheel for Pillow === This error seems unavoidable but it also seems that it doesn't affect the esgf functionality. == References == * [https://github.com/ESGF/esgf-installer/wiki/ESGF-Installation-From-scratch ESGF Installation From Scratch]