# Question about index optimize **URL:** \ **Category:** Elasticsearch\ **Created:** [December 28, 2013, 1:19am UTC](https://discuss.elastic.co/t/question-about-index-optimize/15040 "2013-12-28T01:19:03Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![jackpark](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jackpark](https://discuss.elastic.co/u/jackpark)\ **Post date:** [December 28, 2013, 1:19am UTC](https://discuss.elastic.co/t/question-about-index-optimize/15040/1 "2013-12-28T01:19:03Z")
This page [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html) talks about merging segments. My curiosity/interest in this process grew when, after an overnight data import, the index went down with Too Many Open Files. Sure, I was able to find some instructions about telling nix to set max files to 32000. Question is this: really, how many open files does ES keep when importing monster loads? Would it make sense to do an optimize following a monster load and before further work? If so, from a Java client, how do you do that? Thanks in advance. Jack -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwN%2BXmrBwpCx6fqLAQB3bLJgBwW%3Dv6oHjZ9ZRXdNmnc5Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwN%2BXmrBwpCx6fqLAQB3bLJgBwW%3Dv6oHjZ9ZRXdNmnc5Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jackpark](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jackpark](https://discuss.elastic.co/u/jackpark)\ **Post date:** [December 28, 2013, 3:25am UTC](https://discuss.elastic.co/t/question-about-index-optimize/15040/2 "2013-12-28T03:25:05Z")
After a rather amazing run, I got the "too many open files" report again. The platform is a commodity pc with 8gb ram, 1tb hard disk, running a slightly out of date Ubuntu. I added lines to /etc/security/limits.conf; something about max and min 32000 and booted with ./elasticsearch -f -Xmx4g -Xms2g -Des.index.store.type=niofs -Des.max-open-files=true and still got the crash. I was running the same program on a different platform and, in about the same level of importing, that one blew out with No Node Available; looking at the nix console there says No route to host (never mind that it had been running fine on a local gigabit network with no outside interference. I am still interested whether there is some background "too many open files" going on. Thanks in advance for ideas. Cheers Jack Log trace below: Exception in thread "Thread-16" Exception in thread "Thread-2098" org.elasticsea rch.index.engine.IndexFailedEngineException: [vertices][4] Index failed for [cor e#42641.359576] at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.ja va:497) at org.elasticsearch.index.shard.service.InternalIndexShard.index(Intern alIndexShard.java:386) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP rimary(TransportIndexAction.java:212) at org.elasticsearch.action.support.replication.TransportShardReplicatio nOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplic ationOperationAction.java:556) at org.elasticsearch.action.support.replication.TransportShardReplicatio nOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperat ionAction.java:426) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec utor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: /usr/local/lib/elasticsearch-0.90.9/da ta/elasticsearch/nodes/0/indices/vertices/4/index/\_k7z.fdt (Too many open files) On Fri, Dec 27, 2013 at 5:19 PM, Jack Park [jackpark@topicquests.org](mailto:jackpark@topicquests.org) wrote: > This page > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html) > talks about merging segments. > > My curiosity/interest in this process grew when, after an overnight > data import, the index went down with Too Many Open Files. > > Sure, I was able to find some instructions about telling nix to set > max files to 32000. Question is this: really, how many open files does > ES keep when importing monster loads? > > Would it make sense to do an optimize following a monster load and > before further work? If so, from a Java client, how do you do that? > > Thanks in advance. > Jack -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fx32AV0%2B-KEoct251y%3DRRT6Hb2iXqZXdzSf77bvy%3DPxxQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fx32AV0%2B-KEoct251y%3DRRT6Hb2iXqZXdzSf77bvy%3DPxxQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 1, 2014, 7:36pm UTC](https://discuss.elastic.co/t/question-about-index-optimize/15040/3 "2014-01-01T19:36:32Z")
Hey, can you make sure, that your setting regarding open files is actually applied, by using the nodes info API. See > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. curl -XGET '[http://localhost:9200/\_nodes?process](http://localhost:9200/_nodes?process)' check the max\_file\_descriptors parameter. You can also check [http://localhost:9200/\_cluster/stats?process](http://localhost:9200/_cluster/stats?process) for currently open file descriptors... --Alex On Sat, Dec 28, 2013 at 4:25 AM, Jack Park [jackpark@topicquests.org](mailto:jackpark@topicquests.org) wrote: > After a rather amazing run, I got the "too many open files" report again. > > The platform is a commodity pc with 8gb ram, 1tb hard disk, running a > slightly out of date Ubuntu. > > I added lines to /etc/security/limits.conf; something about max and > min 32000 and booted with ./elasticsearch -f -Xmx4g -Xms2g > -Des.index.store.type=niofs -Des.max-open-files=true > > and still got the crash. > I was running the same program on a different platform and, in about > the same level of importing, that one blew out with No Node Available; > looking at the nix console there says No route to host (never mind > that it had been running fine on a local gigabit network with no > outside interference. > > I am still interested whether there is some background "too many open > files" going on. > > Thanks in advance for ideas. > > Cheers > Jack > > Log trace below: > Exception in thread "Thread-16" Exception in thread "Thread-2098" > org.elasticsea > rch.index.engine.IndexFailedEngineException: [vertices][4] Index failed > for [cor > e#42641.359576] > at > org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.ja > va:497) > at > org.elasticsearch.index.shard.service.InternalIndexShard.index(Intern > alIndexShard.java:386) > at > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > rimary(TransportIndexAction.java:212) > at > org.elasticsearch.action.support.replication.TransportShardReplicatio > > nOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplic > ationOperationAction.java:556) > at > org.elasticsearch.action.support.replication.TransportShardReplicatio > > nOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperat > ionAction.java:426) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec > utor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor > .java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.FileNotFoundException: > /usr/local/lib/elasticsearch-0.90.9/da > ta/elasticsearch/nodes/0/indices/vertices/4/index/\_k7z.fdt (Too many open > files) > > On Fri, Dec 27, 2013 at 5:19 PM, Jack Park [jackpark@topicquests.org](mailto:jackpark@topicquests.org) > wrote: > > > This page > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html) > > > talks about merging segments. > > > > My curiosity/interest in this process grew when, after an overnight > > data import, the index went down with Too Many Open Files. > > > > Sure, I was able to find some instructions about telling nix to set > > max files to 32000. Question is this: really, how many open files does > > ES keep when importing monster loads? > > > > Would it make sense to do an optimize following a monster load and > > before further work? If so, from a Java client, how do you do that? > > > > Thanks in advance. > > Jack > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fx32AV0%2B-KEoct251y%3DRRT6Hb2iXqZXdzSf77bvy%3DPxxQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fx32AV0%2B-KEoct251y%3DRRT6Hb2iXqZXdzSf77bvy%3DPxxQ%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-x3Q8ZWVqsS99bhy6Mrq%2BPFpSFZ-qQAQ6tqd\_UmVD9mg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-x3Q8ZWVqsS99bhy6Mrq%2BPFpSFZ-qQAQ6tqd_UmVD9mg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:58am UTC](https://discuss.elastic.co/t/question-about-index-optimize/15040/4 "2017-07-06T01:58:48Z")