Changes between Version 126 and Version 127 of ESGFNodeInstallation

Timestamp:

Aug 5, 2014 12:41:47 PM (8 years ago)

Author:

terryk

Comment:

--

ESGFNodeInstallation

@@ -438 +438 @@
 }}}
 
+= Statistics for ENES data nodes =
+
+== Prerequisites ==
+1. Install postgres client 
+{{{
+#!sh
+$ yum install postgresql
+}}}
+
+2. Install bind utils
+{{{
+#!sh
+$ yum install bind-utils
+}}}
+
+== Procedure ==
+
+[[NoteBox(note, The procedure (including all the queries) is reported in the [attachment:is-enes2_statistics.txt] file )]]
+
+
+1) Create a 'stats' directory
+   {{{
+   #!sh
+   $ cd stats
+   }}}
+
+1.1)
+   {{{
+   #!sh
+   $ psql esgcet -U dbsuper -c "select distinct remote_addr from esgf_node_manager.access_logging" > ip_addresses.txt
+   }}}
+
+1.2) Open the file ip_addresses.txt and cleanup the first two rows and the last one at the end of the file
+
+2) 
+{{{
+#!sh
+$ ./geoiplookup0.2.sh > ip_addr_country.txt
+}}}
+
+3) Creation of a support table in the esgf_dashboard namespace
+3.1) 
+{{{
+#!sh
+$ psql esgcet -U dbsuper -c "drop table esgf_dashboard.stats_support_table;"
+}}}
+
+3.2) 
+{{{
+#!sh
+$ psql esgcet -U dbsuper -c "create table esgf_dashboard.stats_support_table(ip varchar(20), country varchar(10), eunoteu varchar(10));"
+}}}
+
+4) 
+{{{
+#!sh
+$ psql esgcet -U dbsuper -e < ip_addr_country.txt
+}}}
+
+5) 
+{{{
+#!sh
+$ psql esgcet -U dbsuper -c "update esgf_dashboard.stats_support_table set eunoteu='EU' where country in ('FR','GB','SI','HR','BE','CZ','GR','NO','IT','ES','DE','FI', 'DK', 'SE', 'CH', 'RS', 'NL', 'ME', 'LU', 'PT', 'LI', 'IE', 'HU', 'EE','BG', 'BA', 'AT', 'AL', 'SK', 'RS', 'SM', 'RO', 'PL', 'MC', 'MT');"
+}}}
+
+6) 
+{{{
+#!sh
+$ psql esgcet -U dbsuper -c "update esgf_dashboard.stats_support_table set eunoteu='NotEU' where eunoteu is NULL;"
+}}}
+
+7) 
+{{{
+#!sh
+$ psql esgcet -U dbsuper -c "create table esgf_dashboard.euips as select distinct ip from esgf_dashboard.stats_support_table where eunoteu='EU';"
+}}}
+
+8) Execute query stats:
+   
+   Generate a query_stats_[NUMBER].sql for each statistics queries in attachment:is-enes2_statistics.txt and execute:
+
+   {{{
+   #!sh
+   $ psql esgcet -U dbsuper -e < query_stats_1.sql > query_stats_1.result
+   }}}
+
+9) Generate a zip 
+{{{
+#!sh
+$ zip -9r node_stats_data.meteo.unican.es.zip stats
+}}}
+
+Notes:
+
+3.1 and 3.2 aim at providing the same results.
+3.1 is the original query for the statistics inherited from IS-ENES1. Since the results were not as we expected, we defined the query 3.2
+3.2 is the new query that basically replaces the regexpression with a simple "joining" condition between the access_logging table (node_manager namespace) and the file_version table (publisher)
+
+The same for 4.1 and 4.2
+
+Both at CMCC and DKRZ 3.2 and 4.2 ran fine. On the contrary 3.1 and 4.1 did not provide the expected statistics.
+
+Concerning query number 5:
+
+Get distinct remote locations (by remote address) who have performed at least 1 download from the data node
+select distinct(remote_addr) from esgf_node_manager.access_logging;
+
+please store the result (list of IPs) into a different file and send it to us.
+Actually, this is already provided by step n°1 ("ip_addresses.txt").
+Please send the "ip_addresses.txt" file to us.
+We'll then try to get a global picture about all the remote locations connecting to the IS-ENES data nodes.
+
 = See Also=
 * [wiki:ESGFDataVisibilityAPI ESGF Data Visibility API]