Posts in category tds

TDS is reporting too many open files

The Thredds data server is reporting too many opened files when many users are accessing concurrently to the opendap service.

This is happenning with the ECOMS data gateway which is publishing virtual datasets (NcML) aggregating a huge number of files. For example, the System4 vistual datasets are a collection of 135k of GRIB files and therefore the potential amount of files for this datasets is 3x the amount files (405k files)

To avoid this the limit on opened files must be increased.

The current limits for a session can be checked as:

[user@oceano ~]# ulimit -Hn
16384
[user@oceano ~]# ulimit -Sn
16384

and for a process the PID is required. For example:

[user@oceano ~]# cat /proc/10051/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             191823               191823               processes
Max open files            16384                16384                files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       191823               191823               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

To increase the limit have to edit the security limits for the user:

[root@oceano ~]# nano /etc/security/limits.conf

and write down the hard and soft limits. For example:

user            soft    nofile           100000
user            hard    nofile           100000

be sure that system wide limits are also increased:

cat /proc/sys/fs/file-max

or that you can edit sysctl properties and reload them:

[root@oceano ~]# nano /etc/sysctl.conf

adding this line

fs.file-max = 500000

and reload the new propertie value:

[root@oceano ~]# sysctl -p
...

and check that the new value is update:

[root@oceano ~]# cat /proc/sys/fs/file-max
500000

and for the user limits the user needs to start a new session a check the new limits:

[user@oceano ~]# ulimit -Hn
100000
[user@oceano ~]# ulimit -Sn
100000

and also we can check that processes use the same limits

[user@oceano TDS5]$ cat /proc/32090/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             191823               191823               processes
Max open files            1000000              1000000              files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       191823               191823               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0