TDS is reporting too many open files

The Thredds data server is reporting too many opened files when many users are accessing concurrently to the opendap service.

This is happenning with the ECOMS data gateway which is publishing virtual datasets (NcML) aggregating a huge number of files. For example, the System4 vistual datasets are a collection of 135k of GRIB files and therefore the potential amount of files for this datasets is 3x the amount files (405k files)

To avoid this the limit on opened files must be increased.

The current limits for a session can be checked as:

[user@oceano ~]# ulimit -Hn
16384
[user@oceano ~]# ulimit -Sn
16384

and for a process the PID is required. For example:

[user@oceano ~]# cat /proc/10051/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             191823               191823               processes
Max open files            16384                16384                files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       191823               191823               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

To increase the limit have to edit the security limits for the user:

[root@oceano ~]# nano /etc/security/limits.conf

and write down the hard and soft limits. For example:

user            soft    nofile           100000
user            hard    nofile           100000

be sure that system wide limits are also increased:

cat /proc/sys/fs/file-max

or that you can edit sysctl properties and reload them:

[root@oceano ~]# nano /etc/sysctl.conf

adding this line

fs.file-max = 500000

and reload the new propertie value:

[root@oceano ~]# sysctl -p
...

and check that the new value is update:

[root@oceano ~]# cat /proc/sys/fs/file-max
500000

and for the user limits the user needs to start a new session a check the new limits:

[user@oceano ~]# ulimit -Hn
100000
[user@oceano ~]# ulimit -Sn
100000

and also we can check that processes use the same limits

[user@oceano TDS5]$ cat /proc/32090/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             191823               191823               processes
Max open files            1000000              1000000              files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       191823               191823               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

Comments

1. Alex John -- 2017-02-17 06:32

Howdy dudes! Astonishing stuff keep it up!! personal persuasive essay topics

2. anonymous -- 2017-02-25 11:02

Thank you I am glad about the encouragement! I love your site, you post outstanding. Denver SEO Agency

3. Crells Ton -- 2017-03-01 08:10

I was searching for decisive information on this topic. The information you have provided in the blog is really important. Denver SEO consultant

4. anonymous -- 2017-03-03 12:27

It’s amazing in support of me to truly have a web site that is valuable meant for my knowledge. point of sale display

5. Anthony -- 2017-03-23 07:44

These are truly amongst the wonderful informative blogs. st neots seo

6. anonymous -- 2017-04-17 12:50

I even have been gifted for hours and I haven't suffered such awesome stuff. lurninsider.org

7. Mathieu -- 2017-04-29 08:45

Wonderful, just what a blog it is! This blog has provided the helpful data to us continue the good work. Ryan Cataldo CPG Financial

8. Adrien -- 2017-05-02 07:32

Quality stuff may be the key to invite the users to visit begin to see the blog site, that’s what this site provides. Ryan Cataldo NJ Criminal Record

9. Adrien -- 2017-05-03 09:30

This text may be value everyone’s attention. How will I learn more? RC NJ Criminal Record

10. anonymous -- 2017-05-05 14:20

Outstanding blog, in my opinion site owners should acquire a great deal out of this blog its very user welcoming. Ryan Cataldo Jupiter

11. anonymous -- 2017-05-11 13:21

I'm dotty for your best article writings and contents auspiciously. Phone Number UK

12. anonymous -- 2017-05-24 14:09

Now this is in actual fact cooperative. It’s very openhanded of you to share this with us. Get More Info

13. anonymous -- 2017-05-26 07:25

I want to say thank to you people for this great and helpful info. Thanks!!! free credit card machines

14. anonymous -- 2017-06-17 12:16

You have done really a superb job with your web site. Marvelous stuff is here to read. tax preparer insurance

15. anonymous -- 2017-07-25 13:52

I would never crave to lose out any chance to look throughout your contents. nationwide wireless internet providers

16. anonymous -- 2017-08-08 08:55

I really love all data that you just provide in your articles. outreach seo

17. anonymous -- 2017-08-30 09:17

I'm delighted to mature so additional functional and informative collection on your web site. best way to get rid of termites

18. ruthd.dominick@… -- 2017-08-31 10:05

I absolutely feel delighted once I realize articles appropriate to my work and my subject. <a href="http://thetipsguru.com/category/diy/">how does ear wax get in your ear</a>

19. how to remove squirrels from roof -- 2017-09-06 12:23

I agree. You have made the nice blogs with the great info in the contents. http://thetipsguru.com/how-to-get-rid-of-squirrels/

20. anonymous -- 2017-09-21 15:09

The stuff you are penning blows out my mind. booking com cebu hotels

21. anonymous -- 2017-09-28 07:55

Your articles and contents are encouraging.

http://comparehotelsusa.my-free.website/

22. anonymous -- 2017-10-04 11:53

These are truly amongst the wonderful informative blogs. http://comparehotelsusa.blog.fc2.com/blog-entry-1.html

23. sofia123jas@… -- 2017-11-30 09:53

I am surprised why other specialized don’t perceive your site I’m greatly cheerful I discovered this.

https://migliorcasinoonlinesicuri.it

24. DavidJes -- 2018-01-18 02:27

[url=http://bit.ly/2r83vv5]Buy and Trade Bitcoin and Altcoins. Top 1 Exchange[/url] =$=

25. DavidJes -- 2018-01-19 16:00

[url=http://bit.ly/2r83vv5]Buy and Trade Bitcoin and Altcoins. Top 1 Exchange[/url] =$=

Add New Comment