# Too many open files **URL:** \ **Category:** Elasticsearch\ **Created:** [November 7, 2013, 2:36pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304 "2013-11-07T14:36:50Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![onthefloorr](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@onthefloorr](https://discuss.elastic.co/u/onthefloorr)\ **Post date:** [November 7, 2013, 2:36pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304/1 "2013-11-07T14:36:50Z")
Hi guys, I have a little problem with elasticsearch.. Despite the fact that I set the number of open files on the system as described on the site : [http://www.elasticsearch.org/tutorials/too-many-open-files/](http://www.elasticsearch.org/tutorials/too-many-open-files/) , I always get the error message (""too many open files"") from elasticsearch. Here is the configuration : cluster.name: xxxxxxx index.cache.field.type: soft index.cache.field.max\_size: 10000 indices.store.throttle.type: merge indices.store.throttle.max\_bytes\_per\_sec: 20mb indices.memory.index\_buffer\_size: 10% index.refresh\_interval: 30 index.translog.flush\_threshold\_ops: 20000 index.store.compress.stored: true threadpool.search.type: fixed threadpool.search.size: 4 threadpool.search.queue\_size: 30 threadpool.bulk.type: fixed threadpool.bulk.size: 4 threadpool.bulk.queue\_size: 30 threadpool.index.type: fixed threadpool.index.size: 4 threadpool.index.queue\_size: 30 thrift.port: 9500 indices.recovery.concurrent\_streams: 4 indices.recovery.max\_bytes\_per\_sec: 40mb cluster.routing.allocation.cluster\_concurrent\_rebalance: 20 indices.cache.filter.size: 100mb * * * Open files limit : elasticsearch@xxxx:/$ ulimit -Sn 1000000 elasticsearch@xxxx:/$ ulimit -Hn 1000000 elasticsearch@xxxx:/$ * * * Here is the log file : I started elasticsearch with option es.max-open-files=true . [2013-11-07 15:01:22,538][INFO][bootstrap] max\_open\_files [65510] [2013-11-07 15:01:23,131][INFO][node] [Wiz Kid] version[0.90.5], pid[15414], build[c8714e8/2013-09-17T12:50:20Z] [2013-11-07 15:01:23,131][INFO][node] [Wiz Kid] initializing ... [2013-11-07 15:01:23,518][INFO][plugins] [Wiz Kid] loaded [transport-thrift], sites [HQ, head] [2013-11-07 15:02:31,792][INFO][node] [Wiz Kid] initialized [2013-11-07 15:02:31,894][INFO][node] [Wiz Kid] starting ... [2013-11-07 15:02:31,911][INFO][thrift] [Wiz Kid] bound on port [9500] [2013-11-07 15:02:32,100][INFO][transport] [Wiz Kid] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/172.31.11.162:9300]} [2013-11-07 15:02:35,129][INFO][cluster.service] [Wiz Kid] new\_master [Wiz Kid][ftUK\_RPoSAqTfFEDYMFGuw][inet[/172.31.11.162:9300]], reason: zen-disco-join (elected\_as\_master) [2013-11-07 15:02:35,141][INFO][discovery] [Wiz Kid] elasticsearch\_logs/ftUK\_RPoSAqTfFEDYMFGuw [2013-11-07 15:02:35,228][INFO][http] [Wiz Kid] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/172.31.11.162:9200]} [2013-11-07 15:02:35,228][INFO][node] [Wiz Kid] started [2013-11-07 15:02:55,523][INFO][gateway] [Wiz Kid] recovered [208] indices into cluster\_state * * * Exception: [2013-11-07 15:00:23,343][DEBUG][action.bulk] [Darkdevil] [logs-2013-11-07][2] failed to execute bulk item (index) index {[logs-2013-11-07][logs][G6KBeJFeRoiwKaLXZUKq-g], source[{"message":"xxxxxxxxxx","host":null,"date":"2013-11-07T13:57:45.917Z"}]} org.elasticsearch.index.engine.CreateFailedEngineException: [logs-2013-11-07][2] Create failed for [logs#G6KBeJFeRoiwKaLXZUKq-g] at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:369) at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:331) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:533) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:418) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.FileNotFoundException: /var/lib/elasticsearch/elasticsearch\_xxxxxxxxx/nodes/0/indices/logs-2013-11-07/2/index/\_1kgc.fdt (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:241) at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:466) at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:288) at org.apache.lucene.store.RateLimitedFSDirectory.createOutput(RateLimitedFSDirectory.java:41) at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:419) at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:409) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:62) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:109) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) at org.apache.lucene.index.StoredFieldsProcessor.initFieldsWriter(StoredFieldsProcessor.java:88) at org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:120) at org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65) at org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:264) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:283) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:432) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1513) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1188) at org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:470) at org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:364) ... 8 more * * * There are 1040 shards . ( 5 shard / index ) . XMS and XMX of elasticsearch were set to 4g. The machine has 8 GB of memory, a Intel Xenon 5560 3GHZ 4 cores CPU. Could anyone advise me how to configure elasticsearh ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 7, 2013, 3:12pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304/2 "2013-11-07T15:12:43Z")
It looks like you run a single node? If so, you have too many shards for a single node. 1040 shards is unusual high. Note that each shard can consume around 100-200 file descriptors while being active in the background, so it is quite possible you have exceeded 65510. Check curl localhost:9200/\_nodes/process?pretty or OS utilities for the number of used file descriptors of the JVM. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![11133](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/11133/32/99919_2.png) [@11133](https://discuss.elastic.co/u/11133)\ **Post date:** [November 7, 2013, 3:17pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304/3 "2013-11-07T15:17:47Z")
On Thu, Nov 7, 2013 at 10:36 PM, onthefloorr [onthefloorr@gmail.com](mailto:onthefloorr@gmail.com) wrote: > [2013-11-07 15:01:22,538][INFO][bootstrap] > max\_open\_files [65510] The settings didn't take effect, a reference config steps blow: [https://gist.github.com/larrytin/7094153#file-ubuntu-L13](https://gist.github.com/larrytin/7094153#file-ubuntu-L13) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![onthefloorr](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@onthefloorr](https://discuss.elastic.co/u/onthefloorr)\ **Post date:** [November 7, 2013, 6:12pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304/4 "2013-11-07T18:12:32Z")
Yes, I using elasticsearch on a single node, and it looks like the number of file descriptors exceeded the maximum allowed limit on this machine ☹ The "\_nodes/process?pretty" writes that the maximum\_file\_descriptor is 65535. It seems the jvm only can use 65535 files at once, or could I get it to use more ? If I can't then I have to add a new node ? or there is an another option ? Thank you in advance for your answer. On Thursday, November 7, 2013 4:12:43 PM UTC+1, Jörg Prante wrote: > It looks like you run a single node? If so, you have too many shards for a > single node. 1040 shards is unusual high. Note that each shard can consume > around 100-200 file descriptors while being active in the background, so it > is quite possible you have exceeded 65510. > > Check curl localhost:9200/\_nodes/process?pretty or OS utilities for the > number of used file descriptors of the JVM. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 7, 2013, 8:55pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304/5 "2013-11-07T20:55:52Z")
Check the "nofile" setting in /etc/security/limits.conf for the maximum limit of number of file descriptors. You can change it with superuser privileges. I do not recommend a file descriptor hard limit higher than in /proc/sys/fs/file-max Yes, you should add nodes if you want to manage a high number of active/open shards. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![onthefloorr](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@onthefloorr](https://discuss.elastic.co/u/onthefloorr)\ **Post date:** [November 8, 2013, 7:26am UTC](https://discuss.elastic.co/t/too-many-open-files/14304/6 "2013-11-08T07:26:56Z")
Thank you for your answer. So then I will buy a new machine 🙂 On Thursday, November 7, 2013 9:55:52 PM UTC+1, Jörg Prante wrote: > Check the "nofile" setting in /etc/security/limits.conf for the maximum > limit of number of file descriptors. You can change it with superuser > privileges. I do not recommend a file descriptor hard limit higher than > in /proc/sys/fs/file-max > > Yes, you should add nodes if you want to manage a high number of > active/open shards. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 8, 2013, 7:32am UTC](https://discuss.elastic.co/t/too-many-open-files/14304/7 "2013-11-08T07:32:50Z")
You can also set number of shards to 1 instead 5 I guess. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 8 nov. 2013 à 08:26, onthefloorr [onthefloorr@gmail.com](mailto:onthefloorr@gmail.com) a écrit : > Thank you for your answer. So then I will buy a new machine 🙂 > > On Thursday, November 7, 2013 9:55:52 PM UTC+1, Jörg Prante wrote: > > > Check the "nofile" setting in /etc/security/limits.conf for the maximum limit of number of file descriptors. You can change it with superuser privileges. I do not recommend a file descriptor hard limit higher than in /proc/sys/fs/file-max > > > > Yes, you should add nodes if you want to manage a high number of active/open shards. > > > > Jörg > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pellyadolfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pellyadolfo/32/1879_2.png) [@pellyadolfo](https://discuss.elastic.co/u/pellyadolfo)\ **Post date:** [January 7, 2014, 6:06pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304/8 "2014-01-07T18:06:23Z")
Hi, my model is quite slow with just about some thousands documents.... I realised that, when opening a ``` node = ``` builder.client(clientOnly).data(!clientOnly).local(local).node(); client = node.client(); from my Java program to ES with such a small model, ES automatically creates 10 sockets. Casually I have 10 shards (?). - Is this the expected behavior? - Can I reduce the number of ES shards dynamically to reduce the number of sockets or should I redeploy my ES install? - By opening other connections I finally get up to 200 simultaneous open sockets and, I am afraid, that, when fetching highlight information, some of the results are randomly being lost. Can this missing results be somehow as a consequence of a too large number of open sockets? Thanks for your pointers. Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4c0a4660-ef70-491d-998f-5ed73c4a9025%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4c0a4660-ef70-491d-998f-5ed73c4a9025%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pellyadolfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pellyadolfo/32/1879_2.png) [@pellyadolfo](https://discuss.elastic.co/u/pellyadolfo)\ **Post date:** [January 7, 2014, 7:47pm UTC](https://discuss.elastic.co/t/too-many-open-files/14304/9 "2014-01-07T19:47:39Z")
I guess, my problem with excessive number of sockets could be also a consequence of having 2 JVM running ES, one embedded in Tomcat, a second embedded in other Java app, as said here: [https://groups.google.com/forum/?hl=en-GB#!topicsearchin/elasticsearch/scale|sort:date|spell:true/elasticsearch/m9IWpGzoLLE](https://groups.google.com/forum/?hl=en-GB#!topicsearchin/elasticsearch/scale%7Csort:date%7Cspell:true/elasticsearch/m9IWpGzoLLE) Is there any experience running an unique embedded ES (as jar files), for example, in tomcat's lib folder, being consumed by several tomcat apps and other standalone apps in different JVMs? Any opinion on this configuration as an starting point? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3ba7b377-9b66-4d8b-ad65-de362318f9f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3ba7b377-9b66-4d8b-ad65-de362318f9f2%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pellyadolfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pellyadolfo/32/1879_2.png) [@pellyadolfo](https://discuss.elastic.co/u/pellyadolfo)\ **Post date:** [January 8, 2014, 4:16am UTC](https://discuss.elastic.co/t/too-many-open-files/14304/10 "2014-01-08T04:16:48Z")
Happily, the problem of missing highlight records looks to be gone by making a config change. - Initially I had 2 ES in 2 different apps (a Tomcat and a standalone) configured equal (both listening for incoming TransportClients requests on port 9300 and both open with client(false)) and a third ES connecting to then opened with new TransportClient() to fetch highlighting info. It looks that this third ES was randomly loosing highlighting records. (?) - What I did to fix it was a configuration change to have only one client(false)) ES listening for TransportClients and 2 new TransportClient()s connecting to it. It looks this change fixes the issue which was some kind of coupling between both client(false)) ESs listening on port 9300. Regards -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fbe72b9f-eeac-4d2b-9545-6851352aa3d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fbe72b9f-eeac-4d2b-9545-6851352aa3d5%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:58am UTC](https://discuss.elastic.co/t/too-many-open-files/14304/11 "2017-07-06T01:58:00Z")