| Version 1 (modified by zequi, 4 years ago) (diff) |
|---|
ESGF Local Node Deployment Tutorial
This page shows how to deploy an ESGF Node that provides data, index, compute and idp services and belongs to the esgf-test federation. The purpose of this node is to test publication and other processes before applying them into production.
UNICAN belongs to the Tier 2 group of nodes so the production node MUST be data only.
This page assumes that command are executed by the root user (or sudo -s).
Index
- Prerequisites
- Previous installation clean up
- Installation from scratch using autoinstaller
- Configuration for publication
- Publish the test dataset
- Publish CORDEX datasets
- Known issues
- References
Prerequisites
- You have to create a globus account - https://www.globusid.org/create
- Open ports: Required open ports
[root@spock ~]# iptables -L Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere tcp dpt:http state NEW ACCEPT tcp -- anywhere anywhere tcp dpt:https state NEW ACCEPT tcp -- 192.168.202.0/24 anywhere tcp dpt:ssh ACCEPT tcp -- 193.144.202.0/24 anywhere tcp dpt:ssh ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:gsiftp ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:7512 ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:50000:51000 ACCEPT icmp -- 192.168.202.0/24 anywhere ACCEPT icmp -- anywhere anywhere icmp echo-request state NEW,RELATED,ESTABLISHED Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain SSHSCAN (0 references) target prot opt source destination
1. Previous installation clean up
Execute esg-node stop in order to stop the current ESGF services (in case they are running).
[root@spock ~]# esg-node stop
EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGGGGGGGGGFFFFFFFFFFFFFFFFFFFFFF
E::::::::::::::::::::E SS:::::::::::::::S GGG::::::::::::GF::::::::::::::::::::F
E::::::::::::::::::::ES:::::SSSSSS::::::S GG:::::::::::::::GF::::::::::::::::::::F
EE::::::EEEEEEEEE::::ES:::::S SSSSSSS G:::::GGGGGGGG::::GFF::::::FFFFFFFFF::::F
E:::::E EEEEEES:::::S G:::::G GGGGGG F:::::F FFFFFF
E:::::E S:::::S G:::::G F:::::F
E::::::EEEEEEEEEE S::::SSSS G:::::G F::::::FFFFFFFFFF
E:::::::::::::::E SS::::::SSSSS G:::::G GGGGGGGGGG F:::::::::::::::F
E:::::::::::::::E SSS::::::::SS G:::::G G::::::::G F:::::::::::::::F
E::::::EEEEEEEEEE SSSSSS::::S G:::::G GGGGG::::G F::::::FFFFFFFFFF
E:::::E S:::::SG:::::G G::::G F:::::F
E:::::E EEEEEE S:::::S G:::::G G::::G F:::::F
EE::::::EEEEEEEE:::::ESSSSSSS S:::::S G:::::GGGGGGGG::::GFF:::::::FF
E::::::::::::::::::::ES::::::SSSSSS:::::S GG:::::::::::::::GF::::::::FF
E::::::::::::::::::::ES:::::::::::::::SS GGG::::::GGG:::GF::::::::FF
EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGG GGGGFFFFFFFFFFF.llnl.gov
Checking that you have root privs on spock.meteo.unican.es... [OK]
Checking requisites...
Using IP: 193.144.184.40
Stopping Globus Services for Data-Node... (GridFTP) stop_globus_services for datanode
Stopping globus-gridftp-server: OK
Stopping dashboard provider...
Shutting down ESGF Information Provider... [OK]
Shutting down : esgfnmdNode Manager process not present. Attempting to wipe PIDFILE and LOCKFILE.
Tomcat (jsvc) process is running...
stop tomcat: /usr/local/tomcat/bin/jsvc -pidfile /var/run/tomcat-jsvc.pid -stop org.apache.catalina.startup.Bootstrap
(please wait)
5 Z tomcat 7111 1 0 80 0 - 0 do_exi 2017 ? 00:24:39 [jsvc] <defunct>
postmaster (pid 30159) is running...
Stopping postgresql service: [ OK ]
Stopping httpd: [ OK ]
Running shutdown hooks...
---------------------------
Running Node Services...
node type: [ data ] (4)
---------------------------
---------------------------
Execute source /usr/local/bin/esg-purge.sh && esg-purge all
Installation from scratch using autoinstaller
The autoinstaller allows the installation of data only and data+idp+compute+index nodes. Any other combinations should be installed manually. See https://github.com/ESGF/esgf-installer/wiki/ESGF-Installation-Using-Autoinstaller.
As root (not sudo!):
cd /usr/local/bin wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/esgf-installer/2.5/esg-bootstrap --no-check-certificate chmod 555 esg-bootstrap ./esg-bootstrap
Edit /usr/local/etc/esg-autoinstall.conf. It should looks like:
#!/usr/bin/expect -f # -*- mode:shell-script -*- # # Configuration file for the esg-autoinstall Expect script. # # If you are opening this file as 'esg-autoinstall.template', either # from /usr/local/etc or from a Git repository, DO NOT MODIFY IT # DIRECTLY. It must be copied to /usr/local/etc/esg-autoinstall.conf # before it will be read. # # THIS FILE CONTAINS PASSWORDS -- ensure that when you copy it you set # the permissions to 600 and the ownership to root, e.g. # chown root /usr/local/etc/esg-autoinstall.conf # chmod 600 /usr/local/etc/esg-autoinstall.conf # ### Local Node Settings ### # Set this to "data" or "data compute" for a data-only node # Set this to "index" for an index-only node set NODETYPE "index idp data compute" # Optional install flags # Possibilites include "--devel" or "--debug" set INSTALLFLAGS "" # Fully qualified domain name of the node set FQDN "spock.meteo.unican.es" # Shortname of the node set SHORTNAME "unican test" # Long descriptive name of the node set LONGNAME "unican test" # Admin password (alphanumeric-only) set ADMINPASS "PASSWORD" # IP address cleared for admin-restricted pages set ADMINIP "0.0.0.0" # Password for low-privilege publisher database account (alphanumeric-only) set DBLOWPASS "PASSWORD" # Globus Online username and password # (you must have set up this account in advance) # If your password contains URL special characters, installation may fail. set GLOBUSUSER "YOUR-GLOBUS-USER" set GLOBUSPASS "YOUR-GLOBUS-PASSWORD" # Organization name set ORGNAME "unican" # Namespace (reverse FQDN, i.e. "gov.llnl") set NAMESPACE "es.unican.meteo.spock" # Peer Group (valid values are usually "esgf-test" and "esgf-prod") set PEERGROUP "esgf-test" # Default Peer (set this to empty for the default peergroup member, or to your own FQDN) set DEFAULTPEER "spock.meteo.unican.es" # Hostname of the node to publish to set PUBLISHNODE "spock.meteo.unican.es" # Use an external IDP Peer? # (Unless you know you need to run your own, put y) set EXTERNALIDP "n" # FQDN of the external IDP Peer set IDPPEER "spock.meteo.unican.es" # E-mail address notifications will be sent as set ADMINEMAIL "ezequiel.cimadevilla@unican.es" # Default answer for installing Globus # Should be be N for upgrade and Y for fresh install set GLOBUS "y" # Default answer for automatic peering # Should be be N for upgrade and Y for fresh install set AUTOMATICPEER "y" set THREDDS "N" ### Special-use variables (you probably don't want to change these) ### # Which install script to run set ESGNODESCRIPT /usr/local/bin/esg-node # Disable timeouts (the installer can take a long time) set timeout -1 # Database connection string # (form: [username]@[host]:[port]/[database]) # Note: all elements are mandatory, and the "postgresql://" header is # automatically prepended by the esg-node script! # Currently, the only allowed database name is "esgcet" ## DO NOT CURRENTLY SET THIS TO ANYTHING BUT "" #set DBSTRING "esg@localhost:5432/esgcet set DBSTRING "" # Is this an externally managed database? # (if DBSTRING isn't pointed at localhost, the answer is yes) # DO NOT CURRENTLY SET THIS TO YES set DBEXTERNAL no # Low-privilege database account (this may need to be "esgcet") set DBLOWUSER "esgcet" # PostgreSQL port number # DO NOT CURRENTLY CHANGE THIS set PGPORT 5432 # Force recreation of SOLR replica indexes set SOLRREINDEX ""
