# Too many files open **URL:** \ **Category:** Elasticsearch\ **Created:** [August 15, 2011, 10:19am UTC](https://discuss.elastic.co/t/too-many-files-open/5162 "2011-08-15T10:19:49Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mkleen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mkleen/32/1694_2.png) [@mkleen](https://discuss.elastic.co/u/mkleen)\ **Post date:** [August 15, 2011, 10:19am UTC](https://discuss.elastic.co/t/too-many-files-open/5162/1 "2011-08-15T10:19:49Z")
Hello, i am running Elasticsearch on a Cent-OS and got the following error after running elastisearch for a couple of days. At the moment I have one single Instance of Elasticsearch running. How can I avoid to have too many files open ? Do i have a wrong setup here ? ``` failed recovery]; nested: ``` EngineCreationFailureException[[4e47da12d50c1f1bceeeb795][1] Failed to open reader on writer]; nested: FileNotFoundException[/var/lib/ elasticsearch/test/nodes/0/indices/4e47da12d50c1f1bceeeb795/1/index/ segments\_1 (Too many open files)]; ]] [2011-08-15 06:02:01,740][WARN][indices.cluster] [Flygirl] [4e47da12d50c1f1bceeeb795][0] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [4e47da12d50c1f1bceeeb795][0] failed recovery at org.elasticsearch.index.gateway.IndexShardGatewayService $1.run(IndexShardGatewayService.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [4e47da12d50c1f1bceeeb795][0] Failed to create engine at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: 251) at org.elasticsearch.index.shard.service.InternalIndexShard.start(InternalIndexShard.java: 254) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: 146) at org.elasticsearch.index.gateway.IndexShardGatewayService $1.run(IndexShardGatewayService.java:179) ... 3 more Caused by: java.io.FileNotFoundException: /var/lib/elasticsearch/test/ nodes/0/indices/4e47da12d50c1f1bceeeb795/0/index/segments\_1 (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput $Descriptor.(SimpleFSDirectory.java:69) at org.apache.lucene.store.SimpleFSDirectory $SimpleFSIndexInput.(SimpleFSDirectory.java:90) at org.apache.lucene.store.NIOFSDirectory $NIOFSIndexInput.(NIOFSDirectory.java:91) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java: 78) at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java: 345) at org.elasticsearch.index.store.support.AbstractStore $StoreDirectory.openInput(AbstractStore.java:356) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:262) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java: 359) at org.apache.lucene.index.SegmentInfos $FindSegmentsFile.run(SegmentInfos.java:750) at org.apache.lucene.index.SegmentInfos $FindSegmentsFile.run(SegmentInfos.java:589) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:355) at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1144) at org.elasticsearch.index.engine.robin.RobinEngine.createWriter(RobinEngine.java: 1242) at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: 249) ... 6 more ``` My configuration is as followed: # Cluster Settings cluster: name: search # Server Address #network : # host : 10.0.0.4 # Paths #path: # logs: /var/log/elasticsearch # data: /var/data/elasticsearch # Gateway Settings #gateway: # recover_after_nodes: 1 # recover_after_time: 5m # expected_nodes: 2 # Index Settings index : number_of_shards : 2 number_of_replicas : 1 analysis : analyzer : filename : tokenizer : letter filter : [standard, lowercase, autocomplete] default : tokenizer : standard filter : [standard, lowercase, autocomplete] filter : autocomplete : type : edgeNGram min_gram : 3 max_gram : 15 side : front ``` I also started elasticsearch from my init.d script with ulimit -n 20000 to limit it to have max. 20000 files open. start() { echo -n $"Starting ${NAME}: " ulimit -n 20000 } Many Thanks, Michael ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 15, 2011, 10:58am UTC](https://discuss.elastic.co/t/too-many-files-open/5162/2 "2011-08-15T10:58:38Z")
Which version are you using? The first thing that I would do, is check what the actual number of open files is for the process, the nodes info API provides that in 0.17. On Mon, Aug 15, 2011 at 1:19 PM, mkleen [mkleen@gmail.com](mailto:mkleen@gmail.com) wrote: > Hello, > > i am running Elasticsearch on a Cent-OS and got the following error > after running elastisearch for a couple of days. At the moment I have > one single Instance of Elasticsearch running. How can I avoid to have > too many files open ? Do i have a wrong setup here ? > > ``` > failed recovery]; nested: > > ``` > > EngineCreationFailureException[[4e47da12d50c1f1bceeeb795][1] Failed to > open reader on writer]; nested: FileNotFoundException[/var/lib/ > elasticsearch/test/nodes/0/indices/4e47da12d50c1f1bceeeb795/1/index/ > segments\_1 (Too many open files)]; ]] > [2011-08-15 06:02:01,740][WARN][indices.cluster] > [Flygirl] > [4e47da12d50c1f1bceeeb795][0] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [4e47da12d50c1f1bceeeb795][0] failed recovery > at org.elasticsearch.index.gateway.IndexShardGatewayService > $1.run(IndexShardGatewayService.java:229) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: > org.elasticsearch.index.engine.EngineCreationFailureException: > [4e47da12d50c1f1bceeeb795][0] Failed to create engine > at > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: > 251) > at > > org.elasticsearch.index.shard.service.InternalIndexShard.start(InternalIndexShard.java: > 254) > at > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: > 146) > at org.elasticsearch.index.gateway.IndexShardGatewayService > $1.run(IndexShardGatewayService.java:179) > ... 3 more > Caused by: java.io.FileNotFoundException: > /var/lib/elasticsearch/test/ > nodes/0/indices/4e47da12d50c1f1bceeeb795/0/index/segments\_1 (Too many > open files) > at java.io.RandomAccessFile.open(Native Method) > at > java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput > $Descriptor.(SimpleFSDirectory.java:69) > at org.apache.lucene.store.SimpleFSDirectory > $SimpleFSIndexInput.(SimpleFSDirectory.java:90) > at org.apache.lucene.store.NIOFSDirectory > $NIOFSIndexInput.(NIOFSDirectory.java:91) > at > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java: > 78) > at > org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java: > 345) > at org.elasticsearch.index.store.support.AbstractStore > $StoreDirectory.openInput(AbstractStore.java:356) > at > org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:262) > at > org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java: > 359) > at org.apache.lucene.index.SegmentInfos > $FindSegmentsFile.run(SegmentInfos.java:750) > at org.apache.lucene.index.SegmentInfos > $FindSegmentsFile.run(SegmentInfos.java:589) > at > org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:355) > at > org.apache.lucene.index.IndexWriter.(IndexWriter.java:1144) > at > > org.elasticsearch.index.engine.robin.RobinEngine.createWriter(RobinEngine.java: > 1242) > at > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: > 249) > ... 6 more > > ``` > My configuration is as followed: > > # Cluster Settings > cluster: > name: search > > # Server Address > #network : > # host : 10.0.0.4 > > # Paths > #path: > # logs: /var/log/elasticsearch > # data: /var/data/elasticsearch > > # Gateway Settings > #gateway: > # recover_after_nodes: 1 > # recover_after_time: 5m > # expected_nodes: 2 > > # Index Settings > index : > number_of_shards : 2 > number_of_replicas : 1 > analysis : > analyzer : > filename : > tokenizer : letter > filter : [standard, lowercase, autocomplete] > default : > tokenizer : standard > filter : [standard, lowercase, autocomplete] > filter : > autocomplete : > type : edgeNGram > min_gram : 3 > max_gram : 15 > side : front > > ``` > > I also started elasticsearch from my init.d script with ulimit -n > 20000 to limit it to have max. 20000 files open. > > start() { > echo -n $"Starting ${NAME}: " > ulimit -n 20000 > > } > > Many Thanks, > > Michael ---
**Author:** ![mkleen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mkleen/32/1694_2.png) [@mkleen](https://discuss.elastic.co/u/mkleen)\ **Post date:** [August 15, 2011, 12:32pm UTC](https://discuss.elastic.co/t/too-many-files-open/5162/3 "2011-08-15T12:32:38Z")
On 15 August 2011 12:58, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Which version are you using? The first thing that I would do, is check what > the actual number of open files is for the process, the nodes info API > provides that in 0.17. I'm using 0.17.2 at the moment. The Node Info API gives me the following feedback back: It just gives me the maximum number of files back, but actually not the current one, or do I call the wrong url ? curl -XGET '[http://localhost:9200/\_cluster/nodes](http://localhost:9200/_cluster/nodes)' { "cluster\_name":"search", "nodes":{ "jbSkF0O3THqAo0i-h0YYdw":{ "name":"Lin Sun", "transport\_address":"inet[/10.58.199.249:9300]", "attributes":{ ``` }, "http_address":"inet[/10.58.199.249:9200]", "os":{ "refresh_interval":5000 }, "process":{ "refresh_interval":5000, "id":20531, "max_file_descriptors":20000 }, "jvm":{ "pid":20531, "version":"1.6.0_20", "vm_name":"OpenJDK 64-Bit Server VM", "vm_version":"19.0-b09", "vm_vendor":"Sun Microsystems Inc.", "start_time":1313402828181, "mem":{ "heap_init":"256mb", "heap_init_in_bytes":268435456, "heap_max":"1019.8mb", "heap_max_in_bytes":1069416448, "non_heap_init":"23.1mb", "non_heap_init_in_bytes":24313856, "non_heap_max":"214mb", "non_heap_max_in_bytes":224395264 } }, "network":{ "refresh_interval":5000 }, "transport":{ "bound_address":"inet[/0:0:0:0:0:0:0:0:9300]", "publish_address":"inet[/10.58.199.249:9300]" } } ``` } } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 15, 2011, 3:53pm UTC](https://discuss.elastic.co/t/too-many-files-open/5162/4 "2011-08-15T15:53:40Z")
To get the current number of open files, use the node stats API. The idea is that the node info API gives you the static information of the node, and stats gives you the values and statistics that can change. On Mon, Aug 15, 2011 at 3:32 PM, Michael Kleen [mkleen@gmail.com](mailto:mkleen@gmail.com) wrote: > On 15 August 2011 12:58, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Which version are you using? The first thing that I would do, is check > > what > > the actual number of open files is for the process, the nodes info API > > provides that in 0.17. > > I'm using 0.17.2 at the moment. The Node Info API gives me the > following feedback back: It just gives me the maximum number of files > back, but actually not the current one, or do I call the wrong url ? > > curl -XGET '[http://localhost:9200/\_cluster/nodes](http://localhost:9200/_cluster/nodes)' > > { > "cluster\_name":"search", > "nodes":{ > "jbSkF0O3THqAo0i-h0YYdw":{ > "name":"Lin Sun", > "transport\_address":"inet[/10.58.199.249:9300]", > "attributes":{ > > ``` > }, > "http_address":"inet[/10.58.199.249:9200]", > "os":{ > "refresh_interval":5000 > }, > "process":{ > "refresh_interval":5000, > "id":20531, > "max_file_descriptors":20000 > }, > "jvm":{ > "pid":20531, > "version":"1.6.0_20", > "vm_name":"OpenJDK 64-Bit Server VM", > "vm_version":"19.0-b09", > "vm_vendor":"Sun Microsystems Inc.", > "start_time":1313402828181, > "mem":{ > "heap_init":"256mb", > "heap_init_in_bytes":268435456, > "heap_max":"1019.8mb", > "heap_max_in_bytes":1069416448, > "non_heap_init":"23.1mb", > "non_heap_init_in_bytes":24313856, > "non_heap_max":"214mb", > "non_heap_max_in_bytes":224395264 > } > }, > "network":{ > "refresh_interval":5000 > }, > "transport":{ > "bound_address":"inet[/0:0:0:0:0:0:0:0:9300]", > "publish_address":"inet[/10.58.199.249:9300]" > } > } > > ``` > > } > } ---
**Author:** ![mkleen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mkleen/32/1694_2.png) [@mkleen](https://discuss.elastic.co/u/mkleen)\ **Post date:** [August 15, 2011, 6:11pm UTC](https://discuss.elastic.co/t/too-many-files-open/5162/5 "2011-08-15T18:11:28Z")
Ok, great. So would be a reasonable number of files open ? The last time i had more than 10.000 files open on a single instance ? Should i use several instances instead ? On 15 August 2011 17:53, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > To get the current number of open files, use the node stats API. The idea is > that the node info API gives you the static information of the node, and > stats gives you the values and statistics that can change. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 15, 2011, 6:34pm UTC](https://discuss.elastic.co/t/too-many-files-open/5162/6 "2011-08-15T18:34:33Z")
How many shards do you have on that single instance? It depends mainly on that (each shard is a lucene index). On Mon, Aug 15, 2011 at 9:11 PM, Michael Kleen [mkleen@gmail.com](mailto:mkleen@gmail.com) wrote: > Ok, great. So would be a reasonable number of files open ? The last > time i had more than 10.000 files open on a single instance ? Should i > use several instances instead ? > > On 15 August 2011 17:53, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > To get the current number of open files, use the node stats API. The idea > > is > > that the node info API gives you the static information of the node, and > > stats gives you the values and statistics that can change. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:56am UTC](https://discuss.elastic.co/t/too-many-files-open/5162/7 "2017-07-06T03:56:54Z")