# Too many open files but nofile set to 256000 **URL:** \ **Category:** Elasticsearch\ **Created:** [August 24, 2012, 6:12pm UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847 "2012-08-24T18:12:47Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Marietta](https://avatars.discourse-cdn.com/v4/letter/m/cdc98d/32.png) [@Marietta](https://discuss.elastic.co/u/Marietta)\ **Post date:** [August 24, 2012, 6:12pm UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/1 "2012-08-24T18:12:47Z")
I am sure this must be obvious, but I am missing it. I have cranked the number of open files in /etc/security/limits.conf to 256000 for soft and hard. Elasticsearch confirms when loading that nofile=256000 After an hour or so, one of the ES servers starts reporting too many open files. This is the setup: 6 Servers 4 Servers running ES with data 2 Servers running ES with no data (just routers) There are two indexes: Index 1 - 10 Shards, 1 Replicas (ie Master + 1 replica) - 15GB Index (30GB inc replica) Index 2 - 1 Shard, 0 Replicas - 525MB I have checked number of open files for ES and its only ever around 50-200. So we should be ok. I have been cranking up the nofile setting for the past few weeks but it makes no difference. Any ideas? -- ---
**Author:** ![limac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/limac/32/3981_2.png) [@limac](https://discuss.elastic.co/u/limac)\ **Post date:** [August 26, 2012, 12:25am UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/2 "2012-08-26T00:25:29Z")
Maybe you should check by cat /proc/es pid/limit |grep file Only set module is always not essential. ## Sent from my Windows Phone From: Marietta Sent: 25/08/2012 2:12 AM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Too many open files but nofile set to 256000 I am sure this must be obvious, but I am missing it. I have cranked the number of open files in /etc/security/limits.conf to 256000 for soft and hard. Elasticsearch confirms when loading that nofile=256000 After an hour or so, one of the ES servers starts reporting too many open files. This is the setup: 6 Servers 4 Servers running ES with data 2 Servers running ES with no data (just routers) There are two indexes: Index 1 - 10 Shards, 1 Replicas (ie Master + 1 replica) - 15GB Index (30GB inc replica) Index 2 - 1 Shard, 0 Replicas - 525MB I have checked number of open files for ES and its only ever around 50-200. So we should be ok. I have been cranking up the nofile setting for the past few weeks but it makes no difference. Any ideas? -- -- ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 27, 2012, 9:10am UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/3 "2012-08-27T09:10:24Z")
Can you double check that the max open files is actually applied using the node info API (with the process flag set). If its set, can you gist the lsof output? On Aug 24, 2012, at 9:12 PM, Marietta [mariettavps@gmail.com](mailto:mariettavps@gmail.com) wrote: > I am sure this must be obvious, but I am missing it. > > I have cranked the number of open files in /etc/security/limits.conf to 256000 for soft and hard. Elasticsearch confirms when loading that nofile=256000 > > After an hour or so, one of the ES servers starts reporting too many open files. > > This is the setup: > > 6 Servers > 4 Servers running ES with data > 2 Servers running ES with no data (just routers) > > There are two indexes: > > Index 1 - 10 Shards, 1 Replicas (ie Master + 1 replica) - 15GB Index (30GB inc replica) > Index 2 - 1 Shard, 0 Replicas - 525MB > > I have checked number of open files for ES and its only ever around 50-200. So we should be ok. I have been cranking up the nofile setting for the past few weeks but it makes no difference. > > Any ideas? > > -- -- ---
**Author:** ![Dan\_Fairs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_fairs/32/1205_2.png) [@Dan\_Fairs](https://discuss.elastic.co/u/Dan_Fairs)\ **Post date:** [August 27, 2012, 4:51pm UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/4 "2012-08-27T16:51:12Z")
> I have cranked the number of open files in /etc/security/limits.conf to 256000 for soft and hard. Elasticsearch confirms when loading that nofile=256000 > > After an hour or so, one of the ES servers starts reporting too many open files. Just looking at my deploy scripts, I have the following in /etc/sysctl.d/60-maxfiles.conf: fs.file-max = 500000 ## Cheers, Dan Dan Fairs | [dan.fairs@gmail.com](mailto:dan.fairs@gmail.com) | @danfairs | [www.fezconsulting.com](http://www.fezconsulting.com) -- ---
**Author:** ![Marietta](https://avatars.discourse-cdn.com/v4/letter/m/cdc98d/32.png) [@Marietta](https://discuss.elastic.co/u/Marietta)\ **Post date:** [August 28, 2012, 3:50am UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/5 "2012-08-28T03:50:03Z")
lsof gist - [https://gist.github.com/3494765](https://gist.github.com/3494765) \_nodes shows: ``` "process" : { "refresh_interval" : 1000, "id" : 13266, "max_file_descriptors" : 256000 ``` What am I missing? -- ---
**Author:** ![AlexLevin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexlevin/32/2667_2.png) [@AlexLevin](https://discuss.elastic.co/u/AlexLevin)\ **Post date:** [October 25, 2012, 10:54pm UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/6 "2012-10-25T22:54:13Z")
I have a similar case on a 3 node elastic cluster with nofile set to 128000 elasticsearch version - 0.19.8 As an example on one of the nodes right now I see that elastic is close to the limit and amount of open files is growing: [root@box01 ~]# lsof -u elastic | wc -l108576 [root@box01 ~]# lsof -u elastic | wc -l 108673 I've tried to compare lsof results for two runs and the difference is a lot of files like : ... > java 22771 elastic \*858r REG 8,2 7004 25368715 > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgc.nrm > java 22771 elastic \*859u REG 8,2 1560 25368719 > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.tis > java 22771 elastic \*860u REG 8,2 411 25368721 > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.frq > java 22771 elastic \*861r REG 8,2 348 25368722 > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.prx > java 22771 elastic \*862r REG 8,2 2252 25368717 > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.fdt > java 22771 elastic \*863r REG 8,2 28 25368718 > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.fdx > java 22771 elastic \*864r REG 8,2 88 25368723 > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.nrm > .... Thanks -- Alex On Monday, August 27, 2012 8:50:03 PM UTC-7, Marietta wrote: > lsof gist - [Marietta - lsof · GitHub](https://gist.github.com/3494765) > > \_nodes shows: > > ``` > "process" : { > "refresh_interval" : 1000, > "id" : 13266, > "max_file_descriptors" : 256000 > > ``` > > What am I missing? -- ---
**Author:** ![Chuck\_McKenzie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chuck_mckenzie/32/19660_2.png) [@Chuck\_McKenzie](https://discuss.elastic.co/u/Chuck_McKenzie)\ **Post date:** [November 3, 2012, 5:24am UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/7 "2012-11-03T05:24:00Z")
I've fixed this by optimizing with max\_num\_segments set low, and avoided it by tweaking my merge rules. On Thursday, October 25, 2012 5:54:13 PM UTC-5, AlexLevin wrote: > I have a similar case on a 3 node elastic cluster with nofile set to > 128000 > > elasticsearch version - 0.19.8 > > As an example on one of the nodes right now I see that elastic is close to > the limit and amount of open files is growing: > > [root@box01 ~]# lsof -u elastic | wc -l108576 > [root@box01 ~]# lsof -u elastic | wc -l > 108673 > > I've tried to compare lsof results for two runs and the difference is a > lot of files like : > ... > > > java 22771 elastic \*858r REG 8,2 7004 > > 25368715 > > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgc.nrm > > java 22771 elastic \*859u REG 8,2 1560 > > 25368719 > > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.tis > > java 22771 elastic \*860u REG 8,2 411 > > 25368721 > > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.frq > > java 22771 elastic \*861r REG 8,2 348 > > 25368722 > > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.prx > > java 22771 elastic \*862r REG 8,2 2252 > > 25368717 > > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.fdt > > java 22771 elastic \*863r REG 8,2 28 > > 25368718 > > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.fdx > > java 22771 elastic \*864r REG 8,2 88 > > 25368723 > > /home/elastic/data/elastic280/nodes/0/indices/logstash-2012.10.25/0/index/\_pgd.nrm > > .... > > Thanks > -- Alex > > On Monday, August 27, 2012 8:50:03 PM UTC-7, Marietta wrote: > > > lsof gist - [Marietta - lsof · GitHub](https://gist.github.com/3494765) > > > > \_nodes shows: > > > > ``` > > "process" : { > > "refresh_interval" : 1000, > > "id" : 13266, > > "max_file_descriptors" : 256000 > > > > ``` > > > > What am I missing? -- ---
**Author:** ![smhdiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smhdiu/32/1808_2.png) [@smhdiu](https://discuss.elastic.co/u/smhdiu)\ **Post date:** [November 15, 2012, 11:12am UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/8 "2012-11-15T11:12:43Z")
helo Marietta did you got any resolution ? On Sunday, 26 August 2012 01:25:29 UTC+1, LiMac wrote: > Maybe you should check by > cat /proc/es pid/limit |grep file > > Only set module is always not essential. > > ## Sent from my Windows Phone > > From: Marietta > Sent: 25/08/2012 2:12 AM > To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > Subject: Too many open files but nofile set to 256000 > > I am sure this must be obvious, but I am missing it. > > I have cranked the number of open files in /etc/security/limits.conf to > 256000 for soft and hard. Elasticsearch confirms when loading that > nofile=256000 > > After an hour or so, one of the ES servers starts reporting too many open > files. > > This is the setup: > > 6 Servers > 4 Servers running ES with data > 2 Servers running ES with no data (just routers) > > There are two indexes: > > Index 1 - 10 Shards, 1 Replicas (ie Master + 1 replica) - 15GB Index (30GB > inc replica) > Index 2 - 1 Shard, 0 Replicas - 525MB > > I have checked number of open files for ES and its only ever around > 50-200. So we should be ok. I have been cranking up the nofile setting for > the past few weeks but it makes no difference. > > Any ideas? > > -- -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [November 16, 2012, 7:54am UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/9 "2012-11-16T07:54:15Z")
For anyone with a problem like this, it may be worth confirming the numbers within elasticsearch as well using the nodes info api: /\_nodes?process gives the max\_file\_descriptors: { - refresh\_interval: 1000, - id: 13919, - max\_file\_descriptors: 25000 } /\_nodes/process/stats gives: open\_file\_descriptors: 516, in the output: On Thursday, 15 November 2012 11:12:43 UTC, mohsin husen wrote: > helo Marietta > did you got any resolution ? > > On Sunday, 26 August 2012 01:25:29 UTC+1, LiMac wrote: > > > Maybe you should check by > > cat /proc/es pid/limit |grep file > > > > Only set module is always not essential. > > > > ## Sent from my Windows Phone > > > > From: Marietta > > Sent: 25/08/2012 2:12 AM > > To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > Subject: Too many open files but nofile set to 256000 > > > > I am sure this must be obvious, but I am missing it. > > > > I have cranked the number of open files in /etc/security/limits.conf to > > 256000 for soft and hard. Elasticsearch confirms when loading that > > nofile=256000 > > > > After an hour or so, one of the ES servers starts reporting too many open > > files. > > > > This is the setup: > > > > 6 Servers > > 4 Servers running ES with data > > 2 Servers running ES with no data (just routers) > > > > There are two indexes: > > > > Index 1 - 10 Shards, 1 Replicas (ie Master + 1 replica) - 15GB Index > > (30GB inc replica) > > Index 2 - 1 Shard, 0 Replicas - 525MB > > > > I have checked number of open files for ES and its only ever around > > 50-200. So we should be ok. I have been cranking up the nofile setting for > > the past few weeks but it makes no difference. > > > > Any ideas? > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/too-many-open-files-but-nofile-set-to-256000/8847/10 "2017-07-06T03:04:01Z")