ESGF Local Node Deployment Tutorial
This page shows how to deploy an ESGF Node that provides data, index, compute and idp services and belongs to the esgf-test federation. The purpose of this node is to test publication and other processes before applying them into production.
UNICAN belongs to the Tier 2 group of nodes so the production node MUST be data only.
This page assumes that command are executed by the root user (or sudo -s).
Index
- Prerequisites
- Previous installation clean up
- Installation from scratch using autoinstaller
- Configuration for publication
- Publish the test dataset
- Publish CORDEX datasets
- Known issues
- References
Prerequisites
- You have to create a globus account - https://www.globusid.org/create
- Open ports: Required open ports
[root@spock ~]# iptables -L Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere tcp dpt:http state NEW ACCEPT tcp -- anywhere anywhere tcp dpt:https state NEW ACCEPT tcp -- 192.168.202.0/24 anywhere tcp dpt:ssh ACCEPT tcp -- 193.144.202.0/24 anywhere tcp dpt:ssh ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:gsiftp ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:7512 ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:50000:51000 ACCEPT icmp -- 192.168.202.0/24 anywhere ACCEPT icmp -- anywhere anywhere icmp echo-request state NEW,RELATED,ESTABLISHED Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain SSHSCAN (0 references) target prot opt source destination
1. Previous installation clean up
Execute esg-node stop in order to stop the current ESGF services (in case they are running).
[root@spock ~]# esg-node stop EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGGGGGGGGGFFFFFFFFFFFFFFFFFFFFFF E::::::::::::::::::::E SS:::::::::::::::S GGG::::::::::::GF::::::::::::::::::::F E::::::::::::::::::::ES:::::SSSSSS::::::S GG:::::::::::::::GF::::::::::::::::::::F EE::::::EEEEEEEEE::::ES:::::S SSSSSSS G:::::GGGGGGGG::::GFF::::::FFFFFFFFF::::F E:::::E EEEEEES:::::S G:::::G GGGGGG F:::::F FFFFFF E:::::E S:::::S G:::::G F:::::F E::::::EEEEEEEEEE S::::SSSS G:::::G F::::::FFFFFFFFFF E:::::::::::::::E SS::::::SSSSS G:::::G GGGGGGGGGG F:::::::::::::::F E:::::::::::::::E SSS::::::::SS G:::::G G::::::::G F:::::::::::::::F E::::::EEEEEEEEEE SSSSSS::::S G:::::G GGGGG::::G F::::::FFFFFFFFFF E:::::E S:::::SG:::::G G::::G F:::::F E:::::E EEEEEE S:::::S G:::::G G::::G F:::::F EE::::::EEEEEEEE:::::ESSSSSSS S:::::S G:::::GGGGGGGG::::GFF:::::::FF E::::::::::::::::::::ES::::::SSSSSS:::::S GG:::::::::::::::GF::::::::FF E::::::::::::::::::::ES:::::::::::::::SS GGG::::::GGG:::GF::::::::FF EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGG GGGGFFFFFFFFFFF.llnl.gov Checking that you have root privs on spock.meteo.unican.es... [OK] Checking requisites... Using IP: 193.144.184.40 Stopping Globus Services for Data-Node... (GridFTP) stop_globus_services for datanode Stopping globus-gridftp-server: OK Stopping dashboard provider... Shutting down ESGF Information Provider... [OK] Shutting down : esgfnmdNode Manager process not present. Attempting to wipe PIDFILE and LOCKFILE. Tomcat (jsvc) process is running... stop tomcat: /usr/local/tomcat/bin/jsvc -pidfile /var/run/tomcat-jsvc.pid -stop org.apache.catalina.startup.Bootstrap (please wait) 5 Z tomcat 7111 1 0 80 0 - 0 do_exi 2017 ? 00:24:39 [jsvc] <defunct> postmaster (pid 30159) is running... Stopping postgresql service: [ OK ] Stopping httpd: [ OK ] Running shutdown hooks... --------------------------- Running Node Services... node type: [ data ] (4) --------------------------- ---------------------------
The following command will recursively delete every thredds_data_root defined in esg.ini. umount the data before removing the ESGF node.
Execute source /usr/local/bin/esg-purge.sh && esg-purge all
Installation from scratch using autoinstaller
The autoinstaller allows the installation of data only and data+idp+compute+index nodes. Any other combinations should be installed manually. See https://github.com/ESGF/esgf-installer/wiki/ESGF-Installation-Using-Autoinstaller.
As root (not sudo!):
cd /usr/local/bin wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/esgf-installer/2.5/esg-bootstrap --no-check-certificate chmod 555 esg-bootstrap ./esg-bootstrap
Edit /usr/local/etc/esg-autoinstall.conf. It should looks like:
#!/usr/bin/expect -f # -*- mode:shell-script -*- # # Configuration file for the esg-autoinstall Expect script. # # If you are opening this file as 'esg-autoinstall.template', either # from /usr/local/etc or from a Git repository, DO NOT MODIFY IT # DIRECTLY. It must be copied to /usr/local/etc/esg-autoinstall.conf # before it will be read. # # THIS FILE CONTAINS PASSWORDS -- ensure that when you copy it you set # the permissions to 600 and the ownership to root, e.g. # chown root /usr/local/etc/esg-autoinstall.conf # chmod 600 /usr/local/etc/esg-autoinstall.conf # ### Local Node Settings ### # Set this to "data" or "data compute" for a data-only node # Set this to "index" for an index-only node set NODETYPE "index idp data compute" # Optional install flags # Possibilites include "--devel" or "--debug" set INSTALLFLAGS "" # Fully qualified domain name of the node set FQDN "spock.meteo.unican.es" # Shortname of the node set SHORTNAME "unican test" # Long descriptive name of the node set LONGNAME "unican test" # Admin password (alphanumeric-only) set ADMINPASS "PASSWORD" # IP address cleared for admin-restricted pages set ADMINIP "0.0.0.0" # Password for low-privilege publisher database account (alphanumeric-only) set DBLOWPASS "PASSWORD" # Globus Online username and password # (you must have set up this account in advance) # If your password contains URL special characters, installation may fail. set GLOBUSUSER "YOUR-GLOBUS-USER" set GLOBUSPASS "YOUR-GLOBUS-PASSWORD" # Organization name set ORGNAME "unican" # Namespace (reverse FQDN, i.e. "gov.llnl") set NAMESPACE "es.unican.meteo.spock" # Peer Group (valid values are usually "esgf-test" and "esgf-prod") set PEERGROUP "esgf-test" # Default Peer (set this to empty for the default peergroup member, or to your own FQDN) set DEFAULTPEER "spock.meteo.unican.es" # Hostname of the node to publish to set PUBLISHNODE "spock.meteo.unican.es" # Use an external IDP Peer? # (Unless you know you need to run your own, put y) set EXTERNALIDP "n" # FQDN of the external IDP Peer set IDPPEER "spock.meteo.unican.es" # E-mail address notifications will be sent as set ADMINEMAIL "ezequiel.cimadevilla@unican.es" # Default answer for installing Globus # Should be be N for upgrade and Y for fresh install set GLOBUS "y" # Default answer for automatic peering # Should be be N for upgrade and Y for fresh install set AUTOMATICPEER "y" set THREDDS "N" ### Special-use variables (you probably don't want to change these) ### # Which install script to run set ESGNODESCRIPT /usr/local/bin/esg-node # Disable timeouts (the installer can take a long time) set timeout -1 # Database connection string # (form: [username]@[host]:[port]/[database]) # Note: all elements are mandatory, and the "postgresql://" header is # automatically prepended by the esg-node script! # Currently, the only allowed database name is "esgcet" ## DO NOT CURRENTLY SET THIS TO ANYTHING BUT "" #set DBSTRING "esg@localhost:5432/esgcet set DBSTRING "" # Is this an externally managed database? # (if DBSTRING isn't pointed at localhost, the answer is yes) # DO NOT CURRENTLY SET THIS TO YES set DBEXTERNAL no # Low-privilege database account (this may need to be "esgcet") set DBLOWUSER "esgcet" # PostgreSQL port number # DO NOT CURRENTLY CHANGE THIS set PGPORT 5432 # Force recreation of SOLR replica indexes set SOLRREINDEX ""
Run esg-autoinstall
If everything went right, you should see the following:
---------esgsaml_auth.conf--------- AUTHSERVICE=https://spock.meteo.unican.es/esg-orp/saml/soap/secure/authorizationService.htm --------------------------------- ******************************* Setting up GridFTP... chroot jail ******************************* /esg/gridftp_root does not exist, making it... Creating chroot jail @ /esg/gridftp_root Finished setting up a chroot dir at /esg/gridftp_root. You may wish to create data directories or use mount --bind datadir /esg/gridftp_root/datadir to link in external directories. Reading ESGINI=[/esg/config/esgcet/esg.ini] for thredds_dataset_roots to mount... mounting [/esg/data] into chroot jail [/esg/gridftp_root/] as [esg_dataroot] * writing sanitized passwd file to [/esg/gridftp_root/etc/passwd] [OK] writing sanitized group file to [/esg/gridftp_root/etc/group] [OK] syncing local certificates into chroot jail... [OK] configuring publisher to use this GridFTP server... [OK] Starting globus-gridftp-server: OK Starting Globus Services for IDP-Node... (MyProxy server) Starting Globus services for gateway myproxy-server process is running... Starting search services... master slave `/usr/local/tomcat/webapps/esg-search/WEB-INF/classes/esg/search/config/facets.properties' -> `/esg/config/facets.properties' Using solr_workdir=/usr/local/src/esgf/workbench/esg/solr-5.5.4 Using solr_install_dir=/usr/local/solr-home/slave-8983 Using solr_data_dir=/esg/solr-index/slave-8983 Using solr_server_dir=/usr/local/solr Using solr_logs_dir=/esg/solr-logs Using esg_dist_url=http://dist.ceda.ac.uk/esgf/dist Starting solr server for slave on port [8983] Starting solr: sudo -H -u solr bash -c source /etc/esg.env; SOLR_INCLUDE=/usr/local/solr-home/slave-8983/solr.in.sh bin/solr start -d /usr/local/solr/server -s /usr/local/solr-home/slave-8983 -p 8983 -a '-Denable.master=true -Denable.slave=true' Waiting up to 30 seconds to see Solr running on port 8983 [/] Started Solr server on port 8983 (pid=21572). Happy searching! . Solr process for slave running on port [8983]... check_solr_process [OK] [OK] Using solr_workdir=/usr/local/src/esgf/workbench/esg/solr-5.5.4 Using solr_install_dir=/usr/local/solr-home/master-8984 Using solr_data_dir=/esg/solr-index/master-8984 Using solr_server_dir=/usr/local/solr Using solr_logs_dir=/esg/solr-logs Using esg_dist_url=http://dist.ceda.ac.uk/esgf/dist Starting solr server for master on port [8984] Starting solr: sudo -H -u solr bash -c source /etc/esg.env; SOLR_INCLUDE=/usr/local/solr-home/master-8984/solr.in.sh bin/solr start -d /usr/local/solr/server -s /usr/local/solr-home/master-8984 -p 8984 -a '-Denable.master=true' Waiting up to 30 seconds to see Solr running on port 8984 [/] Started Solr server on port 8984 (pid=21781). Happy searching! . Solr process for master running on port [8984]... check_solr_process [OK] [OK] httpd (pid 25793) is running... --------------------------- Running Node Services... node type: [ data index idp compute ] (61) --------------------------- postmaste 9649 postgres 4u IPv6 21215449 0t0 TCP [::1]:5432 (LISTEN) postmaste 9649 postgres 5u IPv4 21215450 0t0 TCP 127.0.0.1:5432 (LISTEN) myproxy-s 17469 root 5u IPv4 21371727 0t0 TCP *:7512 (LISTEN) globus-gr 17660 root 13u IPv6 21372283 0t0 TCP *:2811 (LISTEN) jsvc 20413 tomcat 47u IPv6 21477473 0t0 TCP *:8080 (LISTEN) jsvc 20413 tomcat 52u IPv6 21477477 0t0 TCP *:8443 (LISTEN) jsvc 20413 tomcat 56u IPv6 21477482 0t0 TCP *:8223 (LISTEN) java 21572 solr 28u IPv6 21480726 0t0 TCP 127.0.0.1:7983 (LISTEN) java 21572 solr 92u IPv6 21481131 0t0 TCP *:8983 (LISTEN) java 21781 solr 28u IPv6 21482161 0t0 TCP 127.0.0.1:7984 (LISTEN) java 21781 solr 92u IPv6 21482512 0t0 TCP *:8984 (LISTEN) httpd 25793 root 5u IPv6 21303256 0t0 TCP *:80 (LISTEN) --------------------------- Finished!... In order to see if this node has been installed properly you may direct your browser to: http://spock.meteo.unican.es/thredds http://spock.meteo.unican.es/esg-orp http://spock.meteo.unican.es/ http://spock.meteo.unican.es/las Your peer group membership -- : [esgf-test] Your specified "default" peer : [spock.meteo.unican.es] Your specified "index" peer - : [spock.meteo.unican.es] (url = http://spock.meteo.unican.es/) Your specified "idp" peer --- : [spock.meteo.unican.es] (name = SPOCK.METEO.UNICAN.ES) Your temporary certificates have been placed in /etc/tempcerts You can install them by executing this : esg-node --install-keypair /etc/tempcerts/hostcert.pem /etc/tempcerts/hostkey.pem When promped for the chainfile, specify: /etc/tempcerts/cacert.pem [Note: Use UNIX group permissions on /esg/content/thredds/esgcet to enable users to be able to publish thredds catalogs from data therein] %> chgrp -R <appropriate unix group for publishing users> /esg/content/thredds ------------------------------------------------------- Administrators of this node should subscribe to the esgf-node-admins@lists.llnl.gov by sending email to: majordomo@lists.llnl.gov with the body: subscribe esgf-node-admins ------------------------------------------------------- v2.5.17-master-release Writing additional settings to db. If these settings already exist, psql will report an error, but ok to disregard. INSERT 0 1 Node installation is complete.
Known issues
Installation stucks when ansible is gathering facts for installing slcs
If installation of slcs stucks when ansible is gathering facts, you need to add gather_subset=!hardware to /etc/ansible/ansible.cfg.
esgpublish - /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not found
rm /usr/lib64/libstdc++.so.6 && ln -s /usr/local/conda/envs/esgf-pub/lib/libstdc++.so.6.0.24 /usr/lib64/libstdc++.so.6
esgpublish: libcurl.so.4: undefined symbol: libssh2_scp_recv2
Source:
(esgf-pub) [root@spock ~]# esgpublish --service fileservice --map mapfiles/test.test.map --project test --thredds --publish --offline Traceback (most recent call last): File "/usr/local/conda/envs/esgf-pub/bin/esgpublish", line 4, in <module> __import__('pkg_resources').run_script('esgcet==3.2.7', 'esgpublish') File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/pkg_resources/__init__.py", line 750, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1527, in run_script exec(code, namespace, namespace) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/EGG-INFO/scripts/esgpublish", line 13, in <module> from esgcet.publish import extractFromDataset, aggregateVariables, filelistIterator, fnmatchIterator, fnIterator, directoryIterator, \ File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/publish/__init__.py", line 3, in <module> from publish import publishDataset, publishDatasetList, pollDatasetPublicationStatus File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/publish/publish.py", line 6, in <module> from esgcet.config import getHandlerByName, getConfig File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/config/__init__.py", line 9, in <module> from cf_handler import CFHandler File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/config/cf_handler.py", line 8, in <module> import cdtime ImportError: /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/../.././libcurl.so.4: undefined symbol: libssh2_scp_recv2
Solution:
conda install curl=7.49.0
esgpublish: THREDDS requires authentication
Check THREDDS username and password in /esg/config/esgcet/esg.ini
esgpublish: Java ServiceException?: Server returned HTTP response code: 500 for URL: http://localhost:8984/solr/datasets/select/
Symptom:
(esgf-pub) [root@spock ~]# esgpublish --service fileservice --map mapfiles/test.test.map --project test --offline --thredds --publish INFO 2018-01-11 10:40:49,192 Replacing files in dataset: test.test, version 1 INFO 2018-01-11 10:40:49,211 File /esg/data/test/sftlf.nc exists, skipping INFO 2018-01-11 10:40:49,211 Adding file info to database INFO 2018-01-11 10:40:49,279 Writing THREDDS catalog /esg/content/thredds/esgcet/1/test.test.v1.xml INFO 2018-01-11 10:40:49,373 Writing THREDDS ESG master catalog /esg/content/thredds/esgcet/catalog.xml INFO 2018-01-11 10:40:49,380 Reinitializing THREDDS server INFO 2018-01-11 10:41:34,120 Publishing: test.test Traceback (most recent call last): File "/usr/local/conda/envs/esgf-pub/bin/esgpublish", line 4, in <module> __import__('pkg_resources').run_script('esgcet==3.2.7', 'esgpublish') File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/pkg_resources/__init__.py", line 750, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1527, in run_script exec(code, namespace, namespace) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/EGG-INFO/scripts/esgpublish", line 741, in <module> main(sys.argv[1:]) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/EGG-INFO/scripts/esgpublish", line 718, in main pid_connector=pid_connector, project_config_section=project_config_section) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/publish/publish.py", line 345, in publishDatasetList dset, statusId, state, evname, status = publishDataset(datasetName, parentIdent, service, threddsRootURL, session, schema=schema, version=versionno) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/publish/publish.py", line 128, in publishDataset statusId = service.createDataset(parentId, threddsURL, -1, "Published") File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/publish/hessianlib.py", line 426, in __call__ return self._invoker(self._method, args) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/publish/hessianlib.py", line 544, in __invoke return self.parse_response(responseProxy) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet-3.2.7-py2.7.egg/esgcet/publish/hessianlib.py", line 555, in parse_response raise value esgcet.publish.hessianlib.RemoteCallException: Java ServiceException: Server returned HTTP response code: 500 for URL: http://localhost:8984/solr/datasets/select/ at esg.search.publish.impl.PublishingServiceImpl.publish(PublishingServiceImpl.java:76) at esg.search.publish.impl.SecurePublishingServiceImpl.publish(SecurePublishingServiceImpl.java:50) at esg.search.publish.impl.RemotePublishingServiceImpl.createDataset(RemotePublishingServiceImpl.java:95) at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:157) at com.caucho.hessian.server.HessianServlet.service(HessianServlet.java:365) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:48) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:478) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
Check solr logs. Solr index may have not been created properly.