# Merge did not occur automatically **URL:** \ **Category:** Elasticsearch\ **Created:** [July 26, 2012, 4:57pm UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526 "2012-07-26T16:57:33Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\ **Post date:** [July 26, 2012, 4:57pm UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/1 "2012-07-26T16:57:33Z")
Hi, I encountered 'Too many open files' after indexing a little more than 1 billion docs. I use ES 1.19.3, 6 nodes in the cluster, 67 indices, 32 shards / index, 1 replica / index. My /etc/security/limits.conf allows 100,000 file descriptors per proc. Some of elasticsearch processes indeed exceeded this limit. (100040) I use merge policy as default (tiered). According to [http://www.elasticsearch.org/guide/reference/index-modules/merge.html](http://www.elasticsearch.org/guide/reference/index-modules/merge.html), index.merge.policy.segments\_per\_tier is 10. I thought the number of Lucene index segments per elasticsearch index would never exceed this number. So I estimated the number of open files would not exceed (67 indices \* 32 shards \* 2 copies \* 10 Lucene index segments / 6 nodes) + (some other ~ 10,000) ~ 20,000 I did 'lsof' and found there are more than 10 Lucene index segments in many of my indices. Ex) sudo /usr/sbin/lsof | grep \ | grep 'nodes/0/indices//\/.\*.fdt' | wc -l, I got 26 I checked curl -XGET localhost:9200/\_nodes/stats?pretty=true, and found "merges":{"current":0 I think this means no merge was performed, right? So, here's my question: How can I configure elasticsearch to automatically perform merge during indexing? Thank you for your help. ---
**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\ **Post date:** [July 26, 2012, 7:08pm UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/2 "2012-07-26T19:08:32Z")
correction! Incorrect: > (67 indices \* 32 shards \* 2 copies \* 10 Lucene index segments / 6 nodes) + (some other ~ 10,000) ~ 20,000 Correct: ((67 indices \* 32 shards \* 2 copies \* 10 Lucene index segments \* about 10 files per Lucene index) / 6 nodes) + (some other descriptors \< 10,000) \< 20,000 file descriptors ---
**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\ **Post date:** [July 26, 2012, 10:13pm UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/3 "2012-07-26T22:13:58Z")
Sorry, there was a typo! Correct: ((67 indices \* 32 shards \* 2 copies \* 10 Lucene index segments \* about 10 files per Lucene index **segment** ) / 6 nodes) + (some other descriptors \< 10,000) \< 20,000 file descriptors ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 27, 2012, 12:07am UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/4 "2012-07-27T00:07:54Z")
Hi, I won't rely on calculating file descriptors by a dry formula, I would just count evidence. On the ES cluster here (0.19.8) I observe 120-150 open files per shard (with default settings, no segment merge tuning, not optimized). Each of the 3 nodes have 32 shards (4 indexes of 2\*12 shards each, that is 1 replica level) resulting in a total number of open files of 4700-4800 on each node (counted by lsof, at idle time = low query, no indexing). If I take my numbers and count your 6 nodes with 67 indices and 32 shards per index, each node must have 1072 shards (or even 2144 with replica?). That is a whopping lot if you ask me. With my worst case factor of 150, I get roughly over 150.000 file descriptors per node. You are very lucky you did not encounter "too many open files" even earlier! I would recommend to halve the number of shards per node, by doubling the nodes, or halve the indexes, or the shards per index. If that is not on option, go for compound format. Best regards, Jörg On Thursday, July 26, 2012 6:57:33 PM UTC+2, arta wrote: > Hi, > I encountered 'Too many open files' after indexing a little more than 1 > billion docs. > I use ES 1.19.3, 6 nodes in the cluster, 67 indices, 32 shards / index, 1 > replica / index. > My /etc/security/limits.conf allows 100,000 file descriptors per proc. > Some of elasticsearch processes indeed exceeded this limit. (100040) > > I use merge policy as default (tiered). According to > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/merge.html), > index.merge.policy.segments\_per\_tier is 10. > I thought the number of Lucene index segments per elasticsearch index > would > never exceed this number. > So I estimated the number of open files would not exceed > (67 indices \* 32 shards \* 2 copies \* 10 Lucene index segments / 6 nodes) + > (some other ~ 10,000) ~ 20,000 > > I did 'lsof' and found there are more than 10 Lucene index segments in > many > of my indices. > Ex) sudo /usr/sbin/lsof | grep \ | grep > 'nodes/0/indices//\/.\*.fdt' | wc -l, I got 26 > > I checked curl -XGET localhost:9200/\_nodes/stats?pretty=true, and found > "merges":{"current":0 > I think this means no merge was performed, right? > > So, here's my question: > How can I configure elasticsearch to automatically perform merge during > indexing? > > Thank you for your help. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873.html](http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\ **Post date:** [July 27, 2012, 1:03am UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/5 "2012-07-27T01:03:07Z")
Thanks for the reply Jorg, I will eventually increase number of hardware, that's why number of shards is relatively big now. We can't increase the number later on, I had to start with that number from the beginning... As for the number of shards, yes I have 4288 shards total. If I divide it by 6 (current number of nodes), it becomes 67_32_2/6=715 shards per node. Taking your number 150, the number of file descriptors needed will be 107250. The limits.con number (100,000) is an arbitrary number and system limit is much higher. Maybe I need to increase the number, but I want to know how many more is enough. ---
**Author:** ![Chris\_Male](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_male/32/2607_2.png) [@Chris\_Male](https://discuss.elastic.co/u/Chris_Male)\ **Post date:** [July 27, 2012, 7:08am UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/6 "2012-07-27T07:08:37Z")
Hi, It's also worth visualizing that in the 'tiered' merge policy there are multiple 'tiers' of segments. You've configured that there can be 10 segments per tier but since there can multiple tiers you might end up with many more segments. Merging will be occurring while you index when the policy feels it's appropriate but you may very well continue to have many segments in your index (which can actually improve performance). On Friday, July 27, 2012 1:03:07 PM UTC+12, arta wrote: > Thanks for the reply Jorg, > I will eventually increase number of hardware, that's why number of shards > is relatively big now. > We can't increase the number later on, I had to start with that number > from > the beginning... > > As for the number of shards, yes I have 4288 shards total. > If I divide it by 6 (current number of nodes), it becomes 67_32_2/6=715 > shards per node. > Taking your number 150, the number of file descriptors needed will be > 107250. > The limits.con number (100,000) is an arbitrary number and system limit is > much higher. > Maybe I need to increase the number, but I want to know how many more is > enough. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873p4020898.html](http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873p4020898.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\ **Post date:** [July 27, 2012, 4:24pm UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/7 "2012-07-27T16:24:19Z")
Thanks Chris, > Merging will be occurring while you index when the policy feels it's appropriate Is there any information what is the criteria that "the policy feels it's appropriate"? Or any guidance which code I should read would be very helpful. ---
**Author:** ![Chris\_Male](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_male/32/2607_2.png) [@Chris\_Male](https://discuss.elastic.co/u/Chris_Male)\ **Post date:** [July 28, 2012, 4:57am UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/8 "2012-07-28T04:57:17Z")
If you feel comfortable looking through some fairly intense Lucene code then I recommend looking at [http://svn.apache.org/viewvc/lucene/dev/branches/lucene\_solr\_3\_6/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java?revision=1362113&view=markup](http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_6/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java?revision=1362113&view=markup) and [http://svn.apache.org/viewvc/lucene/dev/branches/lucene\_solr\_3\_6/lucene/core/src/java/org/apache/lucene/index/MergePolicy.java?revision=1324898&view=markup](http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_6/lucene/core/src/java/org/apache/lucene/index/MergePolicy.java?revision=1324898&view=markup) There is also quite a nice blog entry about merging at > **[Visualizing Lucene's segment merges](https://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)** > > If you've ever wondered how Lucene picks segments to merge during indexing, it looks something like this: That video displays segment merges... On Saturday, July 28, 2012 4:24:19 AM UTC+12, arta wrote: > Thanks Chris, > > > Merging will be occurring while you index when the policy feels it's > > appropriate > > Is there any information what is the criteria that "the policy feels it's > appropriate"? > Or any guidance which code I should read would be very helpful. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873p4020925.html](http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873p4020925.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [July 28, 2012, 9:39am UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/9 "2012-07-28T09:39:35Z")
May I ask how many documents you have in your index. You are running a lot of shards and I wonder if that is really necessary. Anyway, there is another way of reducing the number of open files in Lucene ie ES. You can use also use the Compound-File-System (CFS) option that forces per-segment files to be packed into a single compound file. This "can" influence your search performance so you should evaluate but reduces the # of open files dramatically. see [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/) --\> index.compound\_format simon On Saturday, July 28, 2012 6:57:17 AM UTC+2, Chris Male wrote: > If you feel comfortable looking through some fairly intense Lucene code > then I recommend looking at > [[Apache-SVN] Contents of /lucene/dev/branches/lucene\_solr\_3\_6/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java](http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_6/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java?revision=1362113&view=markup) > and > [[Apache-SVN] Contents of /lucene/dev/branches/lucene\_solr\_3\_6/lucene/core/src/java/org/apache/lucene/index/MergePolicy.java](http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_6/lucene/core/src/java/org/apache/lucene/index/MergePolicy.java?revision=1324898&view=markup) > > There is also quite a nice blog entry about merging at > [Changing Bits: Visualizing Lucene's segment merges](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html) > > On Saturday, July 28, 2012 4:24:19 AM UTC+12, arta wrote: > > > Thanks Chris, > > > > > Merging will be occurring while you index when the policy feels it's > > > appropriate > > > > Is there any information what is the criteria that "the policy feels it's > > appropriate"? > > Or any guidance which code I should read would be very helpful. > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873p4020925.html](http://elasticsearch-users.115913.n3.nabble.com/merge-did-not-occur-automatically-tp4020873p4020925.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\ **Post date:** [July 30, 2012, 8:24pm UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/10 "2012-07-30T20:24:14Z")
Thanks for your guidance, Chris. I'll read the source code. (Hopefully I can understand them..) Thanks for your input, Simon. I'll take a look and experiment CFS. > May I ask how many documents you have in your index. > I will eventually have around 20 billions of docs indexed. > 20,000,000,000 / 2,000 shards ~ 10 million docs / shard > I wanted to have more shards (in other words, less docs per shard), > but I encountered 'Too many open files' issue even in my experimental run I did a couple of month ago, > so I gave up having more shards. > Maybe I could have increased the number of hardware, but there was a budget issue, too 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:18am UTC](https://discuss.elastic.co/t/merge-did-not-occur-automatically/8526/11 "2017-07-06T03:18:26Z")