# Elasticsearch 1.1.0 - Optimize broken?

**URL:** <https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790>\
**Category:** Elasticsearch\
**Created:** [April 3, 2014, 12:47pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790 "2014-04-03T12:47:32Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![Thibaut\_Britz\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thibaut_britz_2/32/1222_2.png) [@Thibaut\_Britz\_2](https://discuss.elastic.co/u/Thibaut_Britz_2)\
**Post date:** [April 12, 2014, 6:09pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/21 "2014-04-12T18:09:15Z")

</div>

Hi,

We also have this issue in 1.0.2. Per index we have over 1000 segments.  
Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,  
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote:

> For reference I'm also on 1.1.0 but I'm not seeing more segments then I  
> expect. I see an average of ~28 per shard on an index I write to  
> constantly. I don't write all that quickly, \< 50 updates a second.
> 
> Nik
> 
> On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:
> 
> > I managed to reproduce the issue locally, I'm looking into it.
> > 
> > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com) wrote:
> > 
> > > Hello,
> > > 
> > > We are experiencing a related problem with 1.1.0. Segments do not seem  
> > > to merge as they should during indexing. The optimize API does practically  
> > > nothing in terms of lowering the segments count either. The problem  
> > > persists through a cluster restart. The vast amount of segments seem to be  
> > > greatly impact the performance of the cluster, in a very negative way.
> > > 
> > > We currently have 414 million documents across 3 nodes, each shard has  
> > > in average 1200 segments(!).
> > > 
> > > With 1.0.1 we had even more documents, ~650 million, without any segment  
> > > problems. Looking in Marvel we were hovering at around 30-40 segments per  
> > > shard back then.
> > > 
> > > Best Regards,  
> > > Robin
> > > 
> > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > 
> > > > Thanks for reporting this, the behavior is definitely unexpected. I'll  
> > > > test \_optimize on very large numbers of shards to see if I can reproduce  
> > > > the issue.
> > > > 
> > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote:
> > > > 
> > > > > Adrien,
> > > > > 
> > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > 
> > > > > I'm glad I got the index optimized, searches are literally 10-20 times  
> > > > > faster without 1500/segments per shard to deal with. It's awesome.
> > > > > 
> > > > > That said, any thoughts on why the index wasn't merging on its own, or  
> > > > > why optimize was returning prematurely?
> > > > > 
> > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote:
> > > > > 
> > > > > > Hi Adrien,
> > > > > > 
> > > > > > I kept the logs up over the last optimize call, and I did see an  
> > > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but  
> > > > > > I don't think that that caused this exception. The error is essentially as  
> > > > > > follows:
> > > > > > 
> > > > > > netty - Caught exception while handling client http traffic, closing  
> > > > > > connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200]
> > > > > > 
> > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream  
> > > > > > at Channels.write  
> > > > > > at OneToOneEncoder.doEncode  
> > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > at Channels.write  
> > > > > > at AbstractChannel.write  
> > > > > > at NettyHttpChannel.sendResponse  
> > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim  
> > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation  
> > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run
> > > > > > 
> > > > > > Sorry about the crappy stack trace. Still, looks like this might  
> > > > > > point to a problem! The exception fired about an hour after I kicked off  
> > > > > > the optimize. Any thoughts?
> > > > > > 
> > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote:
> > > > > > 
> > > > > > > Hi Adrien,
> > > > > > > 
> > > > > > > I did customize my merge policy, although I did so only because I  
> > > > > > > was so surprised by the number of segments left over after the load. I'm  
> > > > > > > pretty sure the optimize problem was happening before I made this change,  
> > > > > > > but either way here are my settings:
> > > > > > > 
> > > > > > > "index" : {  
> > > > > > > "merge" : {  
> > > > > > > "policy" : {  
> > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > "floor\_segment" : "10mb"  
> > > > > > > },  
> > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > }  
> > > > > > > }
> > > > > > > 
> > > > > > > Not sure whether this set up could be a contributing factor or not.  
> > > > > > > Nothing really jumps out at me in the logs. In fact, when i kick off the  
> > > > > > > optimize, I don't see any logging at all. Should I?
> > > > > > > 
> > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > 
> > > > > > > Thanks!
> > > > > > > 
> > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:
> > > > > > > 
> > > > > > > > Hi Elliott,
> > > > > > > > 
> > > > > > > > 1500 segments per shard is certainly way too much, and it is not  
> > > > > > > > normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > 
> > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > - Does the issue still reproduce if you restart your cluster?
> > > > > > > > 
> > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc)  
> > > > > > > > e/current/index-modules-merge.html
> > > > > > > > 
> > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > 
> > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be  
> > > > > > > > > significantly impacting performance? Have you guys noticed this behavior  
> > > > > > > > > elsewhere?
> > > > > > > > > 
> > > > > > > > > Thanks.
> > > > > > > > > 
> > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote:
> > > > > > > > > 
> > > > > > > > > > Adrian,
> > > > > > > > > > 
> > > > > > > > > > I ran the following command:
> > > > > > > > > > 
> > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > 
> > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs  
> > > > > > > > > > showed "cluster state updated".
> > > > > > > > > > 
> > > > > > > > > > I did have to close my index prior to changing the setting and  
> > > > > > > > > > reopen afterward.
> > > > > > > > > > 
> > > > > > > > > > I've since began another optimize, but again it doesn't look like  
> > > > > > > > > > much is happening. The optimize isn't returning and the total CPU usage on  
> > > > > > > > > > every node is holding at about 2% of a single core. I would copy a  
> > > > > > > > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this  
> > > > > > > > > > isn't possible. I can tell you that refreshes of hot\_threads show vary  
> > > > > > > > > > little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing  
> > > > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > thoughts?
> > > > > > > > > > 
> > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > Did you see a message in the logs confirming that the setting  
> > > > > > > > > > > has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > 
> > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc)  
> > > > > > > > > > > e/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > 
> > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > 
> > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times,  
> > > > > > > > > > > > > > and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was  
> > > > > > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even  
> > > > > > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a  
> > > > > > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a  
> > > > > > > > > > > > > > > fair amount of file IO.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a  
> > > > > > > > > > > > > > > > 4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch  
> > > > > > > > > > > > > > > > on each. I've batch loaded approximately 86 million documents into a  
> > > > > > > > > > > > > > > > single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free memory  
> > > > > > > > > > > > > > > > on each server, I would assume that the entire index has been paged into  
> > > > > > > > > > > > > > > > memory (I had worried about disk performance previously, as we are working  
> > > > > > > > > > > > > > > > in a virtualized environment).
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > A stats query on the index in questions shows that the  
> > > > > > > > > > > > > > > > index is composed of \> 7000 segments. This seemed high to me, but maybe  
> > > > > > > > > > > > > > > > it's appropriate. Regardless, I dispatched an optimize command, but I am  
> > > > > > > > > > > > > > > > not seeing any progress and the command has not returned. Current merges  
> > > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot  
> > > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that  
> > > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm  
> > > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has  
> > > > > > > > > > > > > > > > anyone else had similar issues. This is the second time I have tried to  
> > > > > > > > > > > > > > > > optimize an index since upgrading. I've gotten the same result both time.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5)  
> > > > > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > --  
> > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz)  
> > > > > > > > > > > > > iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > .
> > > > > > > > > > > > > 
> > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > 
> > > > > > > > > > > > --  
> > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS)  
> > > > > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gma  
> > > > > > > > > > > > [il.com](http://il.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > .
> > > > > > > > > > > > 
> > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > 
> > > > > > > > > > > --  
> > > > > > > > > > > Adrien Grand
> > > > > > > > > > > 
> > > > > > > > > > > --  
> > > > > > > > > > > You received this message because you are subscribed to a topic  
> > > > > > > > > > > in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > To unsubscribe from this group and all its topics, send an email  
> > > > > > > > > > > to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP)  
> > > > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > .
> > > > > > > > > > > 
> > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922)  
> > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > .
> > > > > > > > > 
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > Adrien Grand
> > > > > > > 
> > > > > > > --  
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > > msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%  
> > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > .
> > > > > 
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > Adrien Grand
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > Adrien Grand
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [April 12, 2014, 11:06pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/22 "2014-04-12T23:06:21Z")

</div>

I see also exploding numbers in a quick local test on 1.1.0, thousands of  
segments are created, but I'm not sure if this behavior is expected or not.  
But that does not look like it should. It slows down bulk indexing  
significantly.

Issuing an optimize just stalls...

curl '0:9200/test/\_optimize?max\_num\_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5 os\_prio=31  
tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait() [0x000000012c140000]  
java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
at java.lang.Object.wait(Native Method)  
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

- locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2262)
- locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
at  
org.elasticsearch.index.engine.internal.InternalEngine.optimize(InternalEngine.java:998)  
at  
org.elasticsearch.index.shard.service.InternalIndexShard.optimize(InternalIndexShard.java:599)  
at  
org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)  
at  
org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)  
at  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(TransportBroadcastOperationAction.java:226)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz [t.britz@trendiction.com](mailto:t.britz@trendiction.com)wrote:

> Hi,
> 
> We also have this issue in 1.0.2. Per index we have over 1000 segments.  
> Optimize doesn't reduce them.
> 
> Would love to see a fix for this.
> 
> Thanks,  
> Thibaut
> 
> On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com)wrote:
> 
> > For reference I'm also on 1.1.0 but I'm not seeing more segments then I  
> > expect. I see an average of ~28 per shard on an index I write to  
> > constantly. I don't write all that quickly, \< 50 updates a second.
> > 
> > Nik
> > 
> > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> > [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:
> > 
> > > I managed to reproduce the issue locally, I'm looking into it.
> > > 
> > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com)wrote:
> > > 
> > > > Hello,
> > > > 
> > > > We are experiencing a related problem with 1.1.0. Segments do not seem  
> > > > to merge as they should during indexing. The optimize API does practically  
> > > > nothing in terms of lowering the segments count either. The problem  
> > > > persists through a cluster restart. The vast amount of segments seem to be  
> > > > greatly impact the performance of the cluster, in a very negative way.
> > > > 
> > > > We currently have 414 million documents across 3 nodes, each shard has  
> > > > in average 1200 segments(!).
> > > > 
> > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > segments per shard back then.
> > > > 
> > > > Best Regards,  
> > > > Robin
> > > > 
> > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > 
> > > > > Thanks for reporting this, the behavior is definitely unexpected. I'll  
> > > > > test \_optimize on very large numbers of shards to see if I can reproduce  
> > > > > the issue.
> > > > > 
> > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com)
> > > > > 
> > > > > > wrote:
> > > > > 
> > > > > > Adrien,
> > > > > > 
> > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > 
> > > > > > I'm glad I got the index optimized, searches are literally 10-20  
> > > > > > times faster without 1500/segments per shard to deal with. It's awesome.
> > > > > > 
> > > > > > That said, any thoughts on why the index wasn't merging on its own,  
> > > > > > or why optimize was returning prematurely?
> > > > > > 
> > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote:
> > > > > > 
> > > > > > > Hi Adrien,
> > > > > > > 
> > > > > > > I kept the logs up over the last optimize call, and I did see an  
> > > > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but  
> > > > > > > I don't think that that caused this exception. The error is essentially as  
> > > > > > > follows:
> > > > > > > 
> > > > > > > netty - Caught exception while handling client http traffic, closing  
> > > > > > > connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200]
> > > > > > > 
> > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown  
> > > > > > > stream  
> > > > > > > at Channels.write  
> > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > at Channels.write  
> > > > > > > at AbstractChannel.write  
> > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim  
> > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe  
> > > > > > > ration  
> > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run
> > > > > > > 
> > > > > > > Sorry about the crappy stack trace. Still, looks like this might  
> > > > > > > point to a problem! The exception fired about an hour after I kicked off  
> > > > > > > the optimize. Any thoughts?
> > > > > > > 
> > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hi Adrien,
> > > > > > > > 
> > > > > > > > I did customize my merge policy, although I did so only because I  
> > > > > > > > was so surprised by the number of segments left over after the load. I'm  
> > > > > > > > pretty sure the optimize problem was happening before I made this change,  
> > > > > > > > but either way here are my settings:
> > > > > > > > 
> > > > > > > > "index" : {  
> > > > > > > > "merge" : {  
> > > > > > > > "policy" : {  
> > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > },  
> > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > }  
> > > > > > > > }
> > > > > > > > 
> > > > > > > > Not sure whether this set up could be a contributing factor or  
> > > > > > > > not. Nothing really jumps out at me in the logs. In fact, when i kick off  
> > > > > > > > the optimize, I don't see any logging at all. Should I?
> > > > > > > > 
> > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > 
> > > > > > > > Thanks!
> > > > > > > > 
> > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:
> > > > > > > > 
> > > > > > > > > Hi Elliott,
> > > > > > > > > 
> > > > > > > > > 1500 segments per shard is certainly way too much, and it is not  
> > > > > > > > > normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > 
> > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > - Does the issue still reproduce if you restart your cluster?
> > > > > > > > > 
> > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc)  
> > > > > > > > > e/current/index-modules-merge.html
> > > > > > > > > 
> > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be  
> > > > > > > > > > significantly impacting performance? Have you guys noticed this behavior  
> > > > > > > > > > elsewhere?
> > > > > > > > > > 
> > > > > > > > > > Thanks.
> > > > > > > > > > 
> > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote:
> > > > > > > > > > 
> > > > > > > > > > > Adrian,
> > > > > > > > > > > 
> > > > > > > > > > > I ran the following command:
> > > > > > > > > > > 
> > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > 
> > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs  
> > > > > > > > > > > showed "cluster state updated".
> > > > > > > > > > > 
> > > > > > > > > > > I did have to close my index prior to changing the setting and  
> > > > > > > > > > > reopen afterward.
> > > > > > > > > > > 
> > > > > > > > > > > I've since began another optimize, but again it doesn't look  
> > > > > > > > > > > like much is happening. The optimize isn't returning and the total CPU  
> > > > > > > > > > > usage on every node is holding at about 2% of a single core. I would copy  
> > > > > > > > > > > a hot\_threads stack trace, but I'm unfortunately on a closed network and  
> > > > > > > > > > > this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing  
> > > > > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > thoughts?
> > > > > > > > > > > 
> > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Did you see a message in the logs confirming that the setting  
> > > > > > > > > > > > has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > 
> > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > 
> > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times,  
> > > > > > > > > > > > > > > and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was  
> > > > > > > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even  
> > > > > > > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a  
> > > > > > > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a  
> > > > > > > > > > > > > > > > fair amount of file IO.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a  
> > > > > > > > > > > > > > > > > 4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch  
> > > > > > > > > > > > > > > > > on each. I've batch loaded approximately 86 million documents into a  
> > > > > > > > > > > > > > > > > single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free memory  
> > > > > > > > > > > > > > > > > on each server, I would assume that the entire index has been paged into  
> > > > > > > > > > > > > > > > > memory (I had worried about disk performance previously, as we are working  
> > > > > > > > > > > > > > > > > in a virtualized environment).
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the  
> > > > > > > > > > > > > > > > > index is composed of \> 7000 segments. This seemed high to me, but maybe  
> > > > > > > > > > > > > > > > > it's appropriate. Regardless, I dispatched an optimize command, but I am  
> > > > > > > > > > > > > > > > > not seeing any progress and the command has not returned. Current merges  
> > > > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot  
> > > > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that  
> > > > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm  
> > > > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has  
> > > > > > > > > > > > > > > > > anyone else had similar issues. This is the second time I have tried to  
> > > > > > > > > > > > > > > > > optimize an index since upgrading. I've gotten the same result both time.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5)  
> > > > > > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz)  
> > > > > > > > > > > > > > iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > .
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > --  
> > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS)  
> > > > > > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gma  
> > > > > > > > > > > > > [il.com](http://il.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > .
> > > > > > > > > > > > > 
> > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > 
> > > > > > > > > > > > --  
> > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > 
> > > > > > > > > > > > --  
> > > > > > > > > > > > You received this message because you are subscribed to a topic  
> > > > > > > > > > > > in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP)  
> > > > > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > .
> > > > > > > > > > > > 
> > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > 
> > > > > > > > > > > --  
> > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922)  
> > > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > .
> > > > > > > > > > 
> > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > Adrien Grand
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > > > msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%  
> > > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > .
> > > > > > 
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > Adrien Grand
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .
> > > > 
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > Adrien Grand
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQSMMGVkb%3DhQhTFTGJiAVcbeig%2B0rEfJutscoMyXtgtQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQSMMGVkb%3DhQhTFTGJiAVcbeig%2B0rEfJutscoMyXtgtQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Robin\_Wallin](https://avatars.discourse-cdn.com/v4/letter/r/3be4f8/32.png) [@Robin\_Wallin](https://discuss.elastic.co/u/Robin_Wallin)\
**Post date:** [April 14, 2014, 6:56am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/23 "2014-04-14T06:56:30Z")

</div>

Yes, that 's the exact same behavior we're seeing. Bulk indexing gradually  
slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize call:  
every 30th minute there are some I/O operations, according to Marvel, and  
the total segment count, which currently is around 280 000, gets lowered by  
about 120-150 segments, then after another 30 minutes it's lowered again  
with about the same amount of segments. This has been going on ever since  
we first issued the optimize API call after the cluster restart. The  
cluster is not serving any search requests or doing any indexing at the  
time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

> I see also exploding numbers in a quick local test on 1.1.0, thousands of  
> segments are created, but I'm not sure if this behavior is expected or not.  
> But that does not look like it should. It slows down bulk indexing  
> significantly.
> 
> Issuing an optimize just stalls...
> 
> curl '0:9200/test/\_optimize?max\_num\_segments=1'
> 
> Does anyone see something similar?
> 
> Stack trace excerpt:
> 
> "elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5  
> os\_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()  
> [0x000000012c140000]  
> java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
> at java.lang.Object.wait(Native Method)  
> at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)
> 
> - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2262)
> - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> at  
> org.elasticsearch.index.engine.internal.InternalEngine.optimize(InternalEngine.java:998)  
> at  
> org.elasticsearch.index.shard.service.InternalIndexShard.optimize(InternalIndexShard.java:599)  
> at  
> org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)  
> at  
> org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)  
> at  
> org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(TransportBroadcastOperationAction.java:226)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
> at java.lang.Thread.run(Thread.java:744)
> 
> (node name "Doctor Strange" was not my selection btw)
> 
> Jörg
> 
> On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz \<[t.b...@trendiction.com](mailto:t.b...@trendiction.com)\<javascript:\>
> 
> > wrote:
> 
> > Hi,
> > 
> > We also have this issue in 1.0.2. Per index we have over 1000 segments.  
> > Optimize doesn't reduce them.
> > 
> > Would love to see a fix for this.
> > 
> > Thanks,  
> > Thibaut
> > 
> > On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett \<[nik...@gmail.com](mailto:nik...@gmail.com)\<javascript:\>
> > 
> > > wrote:
> > 
> > > For reference I'm also on 1.1.0 but I'm not seeing more segments then I  
> > > expect. I see an average of ~28 per shard on an index I write to  
> > > constantly. I don't write all that quickly, \< 50 updates a second.
> > > 
> > > Nik
> > > 
> > > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com) \<javascript:\>\> wrote:
> > > 
> > > > I managed to reproduce the issue locally, I'm looking into it.
> > > > 
> > > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin \<[walr...@gmail.com](mailto:walr...@gmail.com)\<javascript:\>
> > > > 
> > > > > wrote:
> > > > 
> > > > > Hello,
> > > > > 
> > > > > We are experiencing a related problem with 1.1.0. Segments do not seem  
> > > > > to merge as they should during indexing. The optimize API does practically  
> > > > > nothing in terms of lowering the segments count either. The problem  
> > > > > persists through a cluster restart. The vast amount of segments seem to be  
> > > > > greatly impact the performance of the cluster, in a very negative way.
> > > > > 
> > > > > We currently have 414 million documents across 3 nodes, each shard has  
> > > > > in average 1200 segments(!).
> > > > > 
> > > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > > segments per shard back then.
> > > > > 
> > > > > Best Regards,  
> > > > > Robin
> > > > > 
> > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > > 
> > > > > > Thanks for reporting this, the behavior is definitely unexpected.  
> > > > > > I'll test \_optimize on very large numbers of shards to see if I can  
> > > > > > reproduce the issue.
> > > > > > 
> > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<  
> > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > 
> > > > > > > Adrien,
> > > > > > > 
> > > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > > 
> > > > > > > I'm glad I got the index optimized, searches are literally 10-20  
> > > > > > > times faster without 1500/segments per shard to deal with. It's awesome.
> > > > > > > 
> > > > > > > That said, any thoughts on why the index wasn't merging on its own,  
> > > > > > > or why optimize was returning prematurely?
> > > > > > > 
> > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hi Adrien,
> > > > > > > > 
> > > > > > > > I kept the logs up over the last optimize call, and I did see an  
> > > > > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but  
> > > > > > > > I don't think that that caused this exception. The error is essentially as  
> > > > > > > > follows:
> > > > > > > > 
> > > > > > > > netty - Caught exception while handling client http traffic,  
> > > > > > > > closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /  
> > > > > > > > 127.0.0.1:9200]
> > > > > > > > 
> > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown  
> > > > > > > > stream  
> > > > > > > > at Channels.write  
> > > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > at Channels.write  
> > > > > > > > at AbstractChannel.write  
> > > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim  
> > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe  
> > > > > > > > ration  
> > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run
> > > > > > > > 
> > > > > > > > Sorry about the crappy stack trace. Still, looks like this might  
> > > > > > > > point to a problem! The exception fired about an hour after I kicked off  
> > > > > > > > the optimize. Any thoughts?
> > > > > > > > 
> > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi Adrien,
> > > > > > > > > 
> > > > > > > > > I did customize my merge policy, although I did so only because I  
> > > > > > > > > was so surprised by the number of segments left over after the load. I'm  
> > > > > > > > > pretty sure the optimize problem was happening before I made this change,  
> > > > > > > > > but either way here are my settings:
> > > > > > > > > 
> > > > > > > > > "index" : {  
> > > > > > > > > "merge" : {  
> > > > > > > > > "policy" : {  
> > > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > > },  
> > > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > > }  
> > > > > > > > > }
> > > > > > > > > 
> > > > > > > > > Not sure whether this set up could be a contributing factor or  
> > > > > > > > > not. Nothing really jumps out at me in the logs. In fact, when i kick off  
> > > > > > > > > the optimize, I don't see any logging at all. Should I?
> > > > > > > > > 
> > > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > > 
> > > > > > > > > Thanks!
> > > > > > > > > 
> > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:
> > > > > > > > > 
> > > > > > > > > > Hi Elliott,
> > > > > > > > > > 
> > > > > > > > > > 1500 segments per shard is certainly way too much, and it is not  
> > > > > > > > > > normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > > 
> > > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > > - Does the issue still reproduce if you restart your cluster?
> > > > > > > > > > 
> > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc)  
> > > > > > > > > > e/current/index-modules-merge.html
> > > > > > > > > > 
> > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be  
> > > > > > > > > > > significantly impacting performance? Have you guys noticed this behavior  
> > > > > > > > > > > elsewhere?
> > > > > > > > > > > 
> > > > > > > > > > > Thanks.
> > > > > > > > > > > 
> > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Adrian,
> > > > > > > > > > > > 
> > > > > > > > > > > > I ran the following command:
> > > > > > > > > > > > 
> > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > > 
> > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs  
> > > > > > > > > > > > showed "cluster state updated".
> > > > > > > > > > > > 
> > > > > > > > > > > > I did have to close my index prior to changing the setting and  
> > > > > > > > > > > > reopen afterward.
> > > > > > > > > > > > 
> > > > > > > > > > > > I've since began another optimize, but again it doesn't look  
> > > > > > > > > > > > like much is happening. The optimize isn't returning and the total CPU  
> > > > > > > > > > > > usage on every node is holding at about 2% of a single core. I would copy  
> > > > > > > > > > > > a hot\_threads stack trace, but I'm unfortunately on a closed network and  
> > > > > > > > > > > > this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing  
> > > > > > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > > thoughts?
> > > > > > > > > > > > 
> > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Did you see a message in the logs confirming that the setting  
> > > > > > > > > > > > > has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more  
> > > > > > > > > > > > > > > > times, and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was  
> > > > > > > > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even  
> > > > > > > > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a  
> > > > > > > > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a  
> > > > > > > > > > > > > > > > > fair amount of file IO.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got  
> > > > > > > > > > > > > > > > > > a 4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch  
> > > > > > > > > > > > > > > > > > on each. I've batch loaded approximately 86 million documents into a  
> > > > > > > > > > > > > > > > > > single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free memory  
> > > > > > > > > > > > > > > > > > on each server, I would assume that the entire index has been paged into  
> > > > > > > > > > > > > > > > > > memory (I had worried about disk performance previously, as we are working  
> > > > > > > > > > > > > > > > > > in a virtualized environment).
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the  
> > > > > > > > > > > > > > > > > > index is composed of \> 7000 segments. This seemed high to me, but maybe  
> > > > > > > > > > > > > > > > > > it's appropriate. Regardless, I dispatched an optimize command, but I am  
> > > > > > > > > > > > > > > > > > not seeing any progress and the command has not returned. Current merges  
> > > > > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot  
> > > > > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that  
> > > > > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm  
> > > > > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has  
> > > > > > > > > > > > > > > > > > anyone else had similar issues. This is the second time I have tried to  
> > > > > > > > > > > > > > > > > > optimize an index since upgrading. I've gotten the same result both time.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-](https://groups.google.com/d/msgid/elasticsearch/5391291f-)  
> > > > > > > > > > > > > > > > > 5c5e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz)  
> > > > > > > > > > > > > > > iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS)  
> > > > > > > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gma  
> > > > > > > > > > > > > > [il.com](http://il.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > .
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > --  
> > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > 
> > > > > > > > > > > > > --  
> > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP)  
> > > > > > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > .
> > > > > > > > > > > > > 
> > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > 
> > > > > > > > > > > > --  
> > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922)  
> > > > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > .
> > > > > > > > > > > 
> > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > Adrien Grand
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-](https://groups.google.com/d/msgid/elasticsearch/344b09db-)  
> > > > > > > > > a2d8-4c2d-a917-dbf53eda03ce%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > .
> > > > > > > 
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > 
> > > > > > --  
> > > > > > Adrien Grand
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > > > To view this discussion on the web visit  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .
> > > > > 
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > Adrien Grand
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > .
> > > > 
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [April 14, 2014, 8:57am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/24 "2014-04-14T08:57:09Z")

</div>

We found the cause for this issue[1]. There was an issue with the way that  
merges were picked, resulting in too many segments per shard. The issue is  
mostly reproducible with high indexing rates (since it also implies a high  
rate of segment creation) and merge throttling on (the default). We are  
working on publishing a new bugfix release that includes a fix for this.

[1] [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779)  
[https://github.com/elasticsearch/elasticsearch/issues/5779](https://github.com/elasticsearch/elasticsearch/issues/5779)

On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com) wrote:

> Yes, that 's the exact same behavior we're seeing. Bulk indexing gradually  
> slows down to about 1/5th of its usual pace.
> 
> I noted something that might be another clue. During an optimize call:  
> every 30th minute there are some I/O operations, according to Marvel, and  
> the total segment count, which currently is around 280 000, gets lowered by  
> about 120-150 segments, then after another 30 minutes it's lowered again  
> with about the same amount of segments. This has been going on ever since  
> we first issued the optimize API call after the cluster restart. The  
> cluster is not serving any search requests or doing any indexing at the  
> time, so can't be that.
> 
> Robin
> 
> On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:
> 
> > I see also exploding numbers in a quick local test on 1.1.0, thousands of  
> > segments are created, but I'm not sure if this behavior is expected or not.  
> > But that does not look like it should. It slows down bulk indexing  
> > significantly.
> > 
> > Issuing an optimize just stalls...
> > 
> > curl '0:9200/test/\_optimize?max\_num\_segments=1'
> > 
> > Does anyone see something similar?
> > 
> > Stack trace excerpt:
> > 
> > "elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5  
> > os\_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()  
> > [0x000000012c140000]  
> > java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
> > at java.lang.Object.wait(Native Method)  
> > at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)
> > 
> > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > at org.apache.lucene.index.IndexWriter.waitForMerges(  
> > IndexWriter.java:2262)
> > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > at org.elasticsearch.index.engine.internal.InternalEngine.optimize(  
> > InternalEngine.java:998)  
> > at org.elasticsearch.index.shard.service.InternalIndexShard.  
> > optimize(InternalIndexShard.java:599)  
> > at org.elasticsearch.action.admin.indices.optimize.  
> > TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)  
> > at org.elasticsearch.action.admin.indices.optimize.  
> > TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)  
> > at org.elasticsearch.action.support.broadcast.  
> > TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(  
> > TransportBroadcastOperationAction.java:226)  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(  
> > ThreadPoolExecutor.java:1142)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(  
> > ThreadPoolExecutor.java:617)  
> > at java.lang.Thread.run(Thread.java:744)
> > 
> > (node name "Doctor Strange" was not my selection btw)
> > 
> > Jörg
> > 
> > On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz [t.b...@trendiction.com](mailto:t.b...@trendiction.com)wrote:
> > 
> > > Hi,
> > > 
> > > We also have this issue in 1.0.2. Per index we have over 1000 segments.  
> > > Optimize doesn't reduce them.
> > > 
> > > Would love to see a fix for this.
> > > 
> > > Thanks,  
> > > Thibaut
> > > 
> > > On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com)wrote:
> > > 
> > > > For reference I'm also on 1.1.0 but I'm not seeing more segments then I  
> > > > expect. I see an average of ~28 per shard on an index I write to  
> > > > constantly. I don't write all that quickly, \< 50 updates a second.
> > > > 
> > > > Nik
> > > > 
> > > > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<adrien...@elasticsearch.  
> > > > com\> wrote:
> > > > 
> > > > > I managed to reproduce the issue locally, I'm looking into it.
> > > > > 
> > > > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > > 
> > > > > > Hello,
> > > > > > 
> > > > > > We are experiencing a related problem with 1.1.0. Segments do not  
> > > > > > seem to merge as they should during indexing. The optimize API does  
> > > > > > practically nothing in terms of lowering the segments count either. The  
> > > > > > problem persists through a cluster restart. The vast amount of segments  
> > > > > > seem to be greatly impact the performance of the cluster, in a very  
> > > > > > negative way.
> > > > > > 
> > > > > > We currently have 414 million documents across 3 nodes, each shard  
> > > > > > has in average 1200 segments(!).
> > > > > > 
> > > > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > > > segments per shard back then.
> > > > > > 
> > > > > > Best Regards,  
> > > > > > Robin
> > > > > > 
> > > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > > > 
> > > > > > > Thanks for reporting this, the behavior is definitely unexpected.  
> > > > > > > I'll test \_optimize on very large numbers of shards to see if I can  
> > > > > > > reproduce the issue.
> > > > > > > 
> > > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<  
> > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > 
> > > > > > > > Adrien,
> > > > > > > > 
> > > > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > > > 
> > > > > > > > I'm glad I got the index optimized, searches are literally 10-20  
> > > > > > > > times faster without 1500/segments per shard to deal with. It's awesome.
> > > > > > > > 
> > > > > > > > That said, any thoughts on why the index wasn't merging on its own,  
> > > > > > > > or why optimize was returning prematurely?
> > > > > > > > 
> > > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi Adrien,
> > > > > > > > > 
> > > > > > > > > I kept the logs up over the last optimize call, and I did see an  
> > > > > > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but  
> > > > > > > > > I don't think that that caused this exception. The error is essentially as  
> > > > > > > > > follows:
> > > > > > > > > 
> > > > > > > > > netty - Caught exception while handling client http traffic,  
> > > > > > > > > closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /  
> > > > > > > > > 127.0.0.1:9200]
> > > > > > > > > 
> > > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown  
> > > > > > > > > stream  
> > > > > > > > > at Channels.write  
> > > > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > at Channels.write  
> > > > > > > > > at AbstractChannel.write  
> > > > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finis  
> > > > > > > > > hHim  
> > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe  
> > > > > > > > > ration  
> > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run
> > > > > > > > > 
> > > > > > > > > Sorry about the crappy stack trace. Still, looks like this might  
> > > > > > > > > point to a problem! The exception fired about an hour after I kicked off  
> > > > > > > > > the optimize. Any thoughts?
> > > > > > > > > 
> > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > > Hi Adrien,
> > > > > > > > > > 
> > > > > > > > > > I did customize my merge policy, although I did so only because I  
> > > > > > > > > > was so surprised by the number of segments left over after the load. I'm  
> > > > > > > > > > pretty sure the optimize problem was happening before I made this change,  
> > > > > > > > > > but either way here are my settings:
> > > > > > > > > > 
> > > > > > > > > > "index" : {  
> > > > > > > > > > "merge" : {  
> > > > > > > > > > "policy" : {  
> > > > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > > > },  
> > > > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > > > }  
> > > > > > > > > > }
> > > > > > > > > > 
> > > > > > > > > > Not sure whether this set up could be a contributing factor or  
> > > > > > > > > > not. Nothing really jumps out at me in the logs. In fact, when i kick off  
> > > > > > > > > > the optimize, I don't see any logging at all. Should I?
> > > > > > > > > > 
> > > > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > > > 
> > > > > > > > > > Thanks!
> > > > > > > > > > 
> > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi Elliott,
> > > > > > > > > > > 
> > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is not  
> > > > > > > > > > > normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > > > 
> > > > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > > > - Does the issue still reproduce if you restart your cluster?
> > > > > > > > > > > 
> > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc)  
> > > > > > > > > > > e/current/index-modules-merge.html
> > > > > > > > > > > 
> > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be  
> > > > > > > > > > > > significantly impacting performance? Have you guys noticed this behavior  
> > > > > > > > > > > > elsewhere?
> > > > > > > > > > > > 
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > > 
> > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Adrian,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I ran the following command:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > > > 
> > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs  
> > > > > > > > > > > > > showed "cluster state updated".
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I did have to close my index prior to changing the setting and  
> > > > > > > > > > > > > reopen afterward.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I've since began another optimize, but again it doesn't look  
> > > > > > > > > > > > > like much is happening. The optimize isn't returning and the total CPU  
> > > > > > > > > > > > > usage on every node is holding at about 2% of a single core. I would copy  
> > > > > > > > > > > > > a hot\_threads stack trace, but I'm unfortunately on a closed network and  
> > > > > > > > > > > > > this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing  
> > > > > > > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Did you see a message in the logs confirming that the setting  
> > > > > > > > > > > > > > has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more  
> > > > > > > > > > > > > > > > > times, and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something  
> > > > > > > > > > > > > > > > > > was happening in the background, but I'm still seeing over 6500 segments.  
> > > > > > > > > > > > > > > > > > Even after setting max\_num\_segments=5. Does this seem right? Queries are  
> > > > > > > > > > > > > > > > > > a little faster (350-400ms) but still not great. Bigdesk is still showing  
> > > > > > > > > > > > > > > > > > a fair amount of file IO.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got  
> > > > > > > > > > > > > > > > > > > a 4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch  
> > > > > > > > > > > > > > > > > > > on each. I've batch loaded approximately 86 million documents into a  
> > > > > > > > > > > > > > > > > > > single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free  
> > > > > > > > > > > > > > > > > > > memory on each server, I would assume that the entire index has been paged  
> > > > > > > > > > > > > > > > > > > into memory (I had worried about disk performance previously, as we are  
> > > > > > > > > > > > > > > > > > > working in a virtualized environment).
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the  
> > > > > > > > > > > > > > > > > > > index is composed of \> 7000 segments. This seemed high to me, but maybe  
> > > > > > > > > > > > > > > > > > > it's appropriate. Regardless, I dispatched an optimize command, but I am  
> > > > > > > > > > > > > > > > > > > not seeing any progress and the command has not returned. Current merges  
> > > > > > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot  
> > > > > > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that  
> > > > > > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm  
> > > > > > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has  
> > > > > > > > > > > > > > > > > > > anyone else had similar issues. This is the second time I have tried to  
> > > > > > > > > > > > > > > > > > > optimize an index since upgrading. I've gotten the same result both time.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)  
> > > > > > > > > > > > > > > > > > .  
> > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-](https://groups.google.com/d/msgid/elasticsearch/5391291f-)  
> > > > > > > > > > > > > > > > > > 5c5e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO  
> > > > > > > > > > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS)  
> > > > > > > > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gma  
> > > > > > > > > > > > > > > [il.com](http://il.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP)  
> > > > > > > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.c  
> > > > > > > > > > > > > > om[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > .
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > --  
> > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922)  
> > > > > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > .
> > > > > > > > > > > > 
> > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > 
> > > > > > > > > > > --  
> > > > > > > > > > > Adrien Grand
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d)  
> > > > > > > > > > 8-4c2d-a917-dbf53eda03ce%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > .
> > > > > > > > 
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > 
> > > > > > > --  
> > > > > > > Adrien Grand
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%  
> > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > .
> > > > > > 
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > Adrien Grand
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-  
> > > > > csXPqtdPad%3DC0QFiTyYOzsU2Bg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > .
> > > > > 
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn  
> > > > 5UO0QNfeHHFGuYENwLD6w%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > .
> > > > 
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQs  
> > > UptE22kDKbB3NMdcA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5MhJ71sdYz7P4o-N-4E161o2ORMQ-vt%2BdmY%2B54FLOKLw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5MhJ71sdYz7P4o-N-4E161o2ORMQ-vt%2BdmY%2B54FLOKLw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Robin\_Wallin](https://avatars.discourse-cdn.com/v4/letter/r/3be4f8/32.png) [@Robin\_Wallin](https://discuss.elastic.co/u/Robin_Wallin)\
**Post date:** [April 14, 2014, 9:20am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/25 "2014-04-14T09:20:51Z")

</div>

Great! Is there anything we can do in the meantime in order to decrease the  
segment count?

Also, I understand that the issue will solve the amount of segments created  
when indexing, but what about optimize? Will it be implicitly fixed as well?

Thanks!

Robin

On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:

> We found the cause for this issue[1]. There was an issue with the way that  
> merges were picked, resulting in too many segments per shard. The issue is  
> mostly reproducible with high indexing rates (since it also implies a high  
> rate of segment creation) and merge throttling on (the default). We are  
> working on publishing a new bugfix release that includes a fix for this.
> 
> [1] [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779)  
> [https://github.com/elasticsearch/elasticsearch/issues/5779](https://github.com/elasticsearch/elasticsearch/issues/5779)
> 
> On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin \<[walr...@gmail.com](mailto:walr...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > Yes, that 's the exact same behavior we're seeing. Bulk indexing  
> > gradually slows down to about 1/5th of its usual pace.
> > 
> > I noted something that might be another clue. During an optimize call:  
> > every 30th minute there are some I/O operations, according to Marvel, and  
> > the total segment count, which currently is around 280 000, gets lowered by  
> > about 120-150 segments, then after another 30 minutes it's lowered again  
> > with about the same amount of segments. This has been going on ever since  
> > we first issued the optimize API call after the cluster restart. The  
> > cluster is not serving any search requests or doing any indexing at the  
> > time, so can't be that.
> > 
> > Robin
> > 
> > On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:
> > 
> > > I see also exploding numbers in a quick local test on 1.1.0, thousands  
> > > of segments are created, but I'm not sure if this behavior is expected or  
> > > not. But that does not look like it should. It slows down bulk indexing  
> > > significantly.
> > > 
> > > Issuing an optimize just stalls...
> > > 
> > > curl '0:9200/test/\_optimize?max\_num\_segments=1'
> > > 
> > > Does anyone see something similar?
> > > 
> > > Stack trace excerpt:
> > > 
> > > "elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5  
> > > os\_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()  
> > > [0x000000012c140000]  
> > > java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
> > > at java.lang.Object.wait(Native Method)  
> > > at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)
> > > 
> > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > > at org.apache.lucene.index.IndexWriter.waitForMerges(  
> > > IndexWriter.java:2262)
> > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > > at org.elasticsearch.index.engine.internal.InternalEngine.optimize(  
> > > InternalEngine.java:998)  
> > > at org.elasticsearch.index.shard.service.InternalIndexShard.  
> > > optimize(InternalIndexShard.java:599)  
> > > at org.elasticsearch.action.admin.indices.optimize.  
> > > TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)  
> > > at org.elasticsearch.action.admin.indices.optimize.  
> > > TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)  
> > > at org.elasticsearch.action.support.broadcast.  
> > > TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(  
> > > TransportBroadcastOperationAction.java:226)  
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(  
> > > ThreadPoolExecutor.java:1142)  
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(  
> > > ThreadPoolExecutor.java:617)  
> > > at java.lang.Thread.run(Thread.java:744)
> > > 
> > > (node name "Doctor Strange" was not my selection btw)
> > > 
> > > Jörg
> > > 
> > > On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz [t.b...@trendiction.com](mailto:t.b...@trendiction.com)wrote:
> > > 
> > > > Hi,
> > > > 
> > > > We also have this issue in 1.0.2. Per index we have over 1000 segments.  
> > > > Optimize doesn't reduce them.
> > > > 
> > > > Would love to see a fix for this.
> > > > 
> > > > Thanks,  
> > > > Thibaut
> > > > 
> > > > On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com)wrote:
> > > > 
> > > > > For reference I'm also on 1.1.0 but I'm not seeing more segments then  
> > > > > I expect. I see an average of ~28 per shard on an index I write to  
> > > > > constantly. I don't write all that quickly, \< 50 updates a second.
> > > > > 
> > > > > Nik
> > > > > 
> > > > > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > 
> > > > > > I managed to reproduce the issue locally, I'm looking into it.
> > > > > > 
> > > > > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > > > 
> > > > > > > Hello,
> > > > > > > 
> > > > > > > We are experiencing a related problem with 1.1.0. Segments do not  
> > > > > > > seem to merge as they should during indexing. The optimize API does  
> > > > > > > practically nothing in terms of lowering the segments count either. The  
> > > > > > > problem persists through a cluster restart. The vast amount of segments  
> > > > > > > seem to be greatly impact the performance of the cluster, in a very  
> > > > > > > negative way.
> > > > > > > 
> > > > > > > We currently have 414 million documents across 3 nodes, each shard  
> > > > > > > has in average 1200 segments(!).
> > > > > > > 
> > > > > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > > > > segments per shard back then.
> > > > > > > 
> > > > > > > Best Regards,  
> > > > > > > Robin
> > > > > > > 
> > > > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > > > > 
> > > > > > > > Thanks for reporting this, the behavior is definitely unexpected.  
> > > > > > > > I'll test \_optimize on very large numbers of shards to see if I can  
> > > > > > > > reproduce the issue.
> > > > > > > > 
> > > > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<  
> > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > 
> > > > > > > > > Adrien,
> > > > > > > > > 
> > > > > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > > > > 
> > > > > > > > > I'm glad I got the index optimized, searches are literally 10-20  
> > > > > > > > > times faster without 1500/segments per shard to deal with. It's awesome.
> > > > > > > > > 
> > > > > > > > > That said, any thoughts on why the index wasn't merging on its  
> > > > > > > > > own, or why optimize was returning prematurely?
> > > > > > > > > 
> > > > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > > Hi Adrien,
> > > > > > > > > > 
> > > > > > > > > > I kept the logs up over the last optimize call, and I did see an  
> > > > > > > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but  
> > > > > > > > > > I don't think that that caused this exception. The error is essentially as  
> > > > > > > > > > follows:
> > > > > > > > > > 
> > > > > > > > > > netty - Caught exception while handling client http traffic,  
> > > > > > > > > > closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /  
> > > > > > > > > > 127.0.0.1:9200]
> > > > > > > > > > 
> > > > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown  
> > > > > > > > > > stream  
> > > > > > > > > > at Channels.write  
> > > > > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > at Channels.write  
> > > > > > > > > > at AbstractChannel.write  
> > > > > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finis  
> > > > > > > > > > hHim  
> > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe  
> > > > > > > > > > ration  
> > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run
> > > > > > > > > > 
> > > > > > > > > > Sorry about the crappy stack trace. Still, looks like this might  
> > > > > > > > > > point to a problem! The exception fired about an hour after I kicked off  
> > > > > > > > > > the optimize. Any thoughts?
> > > > > > > > > > 
> > > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > 
> > > > > > > > > > > I did customize my merge policy, although I did so only because  
> > > > > > > > > > > I was so surprised by the number of segments left over after the load. I'm  
> > > > > > > > > > > pretty sure the optimize problem was happening before I made this change,  
> > > > > > > > > > > but either way here are my settings:
> > > > > > > > > > > 
> > > > > > > > > > > "index" : {  
> > > > > > > > > > > "merge" : {  
> > > > > > > > > > > "policy" : {  
> > > > > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > > > > },  
> > > > > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > > > > }  
> > > > > > > > > > > }
> > > > > > > > > > > 
> > > > > > > > > > > Not sure whether this set up could be a contributing factor or  
> > > > > > > > > > > not. Nothing really jumps out at me in the logs. In fact, when i kick off  
> > > > > > > > > > > the optimize, I don't see any logging at all. Should I?
> > > > > > > > > > > 
> > > > > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > > > > 
> > > > > > > > > > > Thanks!
> > > > > > > > > > > 
> > > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi Elliott,
> > > > > > > > > > > > 
> > > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is  
> > > > > > > > > > > > not normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > > > > 
> > > > > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > > > > - Does the issue still reproduce if you restart your cluster?
> > > > > > > > > > > > 
> > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > reference/current/index-modules-merge.html
> > > > > > > > > > > > 
> > > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be  
> > > > > > > > > > > > > significantly impacting performance? Have you guys noticed this behavior  
> > > > > > > > > > > > > elsewhere?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Adrian,
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I ran the following command:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The  
> > > > > > > > > > > > > > logs showed "cluster state updated".
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I did have to close my index prior to changing the setting  
> > > > > > > > > > > > > > and reopen afterward.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I've since began another optimize, but again it doesn't look  
> > > > > > > > > > > > > > like much is happening. The optimize isn't returning and the total CPU  
> > > > > > > > > > > > > > usage on every node is holding at about 2% of a single core. I would copy  
> > > > > > > > > > > > > > a hot\_threads stack trace, but I'm unfortunately on a closed network and  
> > > > > > > > > > > > > > this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing  
> > > > > > > > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Did you see a message in the logs confirming that the  
> > > > > > > > > > > > > > > setting has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more  
> > > > > > > > > > > > > > > > > > times, and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something  
> > > > > > > > > > > > > > > > > > > was happening in the background, but I'm still seeing over 6500 segments.  
> > > > > > > > > > > > > > > > > > > Even after setting max\_num\_segments=5. Does this seem right? Queries are  
> > > > > > > > > > > > > > > > > > > a little faster (350-400ms) but still not great. Bigdesk is still showing  
> > > > > > > > > > > > > > > > > > > a fair amount of file IO.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've  
> > > > > > > > > > > > > > > > > > > > got a 4 node cluster, each with 64G of ram, with 24G allocated to  
> > > > > > > > > > > > > > > > > > > > Elasticsearch on each. I've batch loaded approximately 86 million  
> > > > > > > > > > > > > > > > > > > > documents into a single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free  
> > > > > > > > > > > > > > > > > > > > memory on each server, I would assume that the entire index has been paged  
> > > > > > > > > > > > > > > > > > > > into memory (I had worried about disk performance previously, as we are  
> > > > > > > > > > > > > > > > > > > > working in a virtualized environment).
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the  
> > > > > > > > > > > > > > > > > > > > index is composed of \> 7000 segments. This seemed high to me, but maybe  
> > > > > > > > > > > > > > > > > > > > it's appropriate. Regardless, I dispatched an optimize command, but I am  
> > > > > > > > > > > > > > > > > > > > not seeing any progress and the command has not returned. Current merges  
> > > > > > > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot  
> > > > > > > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that  
> > > > > > > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm  
> > > > > > > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has  
> > > > > > > > > > > > > > > > > > > > anyone else had similar issues. This is the second time I have tried to  
> > > > > > > > > > > > > > > > > > > > optimize an index since upgrading. I've gotten the same result both time.
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > > > from it, send an email to elasticsearc...@googlegroups.c  
> > > > > > > > > > > > > > > > > > > om.  
> > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-](https://groups.google.com/d/msgid/elasticsearch/5391291f-)  
> > > > > > > > > > > > > > > > > > > 5c5e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO  
> > > > > > > > > > > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%](https://groups.google.com/d/msgid/elasticsearch/CAGCt%25)  
> > > > > > > > > > > > > > > > 2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D  
> > > > > > > > > > > > > > > > emc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/)  
> > > > > > > > > > > > > > > unsubscribe.  
> > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP)  
> > > > > > > > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.c  
> > > > > > > > > > > > > > > om[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922)  
> > > > > > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > .
> > > > > > > > > > > > > 
> > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > 
> > > > > > > > > > > > --  
> > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > 
> > > > > > > > > > > --  
> > > > > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d)  
> > > > > > > > > > > 8-4c2d-a917-dbf53eda03ce%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > .
> > > > > > > > > 
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > Adrien Grand
> > > > > > > 
> > > > > > > --  
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-](https://groups.google.com/d/msgid/elasticsearch/eda52a43-)  
> > > > > > > 94ec-4574-b989-32727cf3cfe4%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > .
> > > > > > > 
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > 
> > > > > > --  
> > > > > > Adrien Grand
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-  
> > > > > > csXPqtdPad%3DC0QFiTyYOzsU2Bg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > .
> > > > > > 
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn  
> > > > > 5UO0QNfeHHFGuYENwLD6w%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > .
> > > > > 
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQs  
> > > > UptE22kDKbB3NMdcA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > .
> > > > 
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [April 14, 2014, 9:43am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/26 "2014-04-14T09:43:56Z")

</div>

Setting `index.merge.scheduler` to `concurrentmergescheduler`[1] (beware  
that this is not a live setting so it requires closing the index) should  
help mitigate the problem.

Indeed, this fix also implicitely fixes the optimize issue as well: I just  
ran again my script that reproduced the optimize issue that Elliott  
described and it doesn't reproduce anymore with the fix.

[1]

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com) wrote:

> Great! Is there anything we can do in the meantime in order to decrease  
> the segment count?
> 
> Also, I understand that the issue will solve the amount of segments  
> created when indexing, but what about optimize? Will it be implicitly fixed  
> as well?
> 
> Thanks!
> 
> Robin
> 
> On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:
> 
> > We found the cause for this issue[1]. There was an issue with the way  
> > that merges were picked, resulting in too many segments per shard. The  
> > issue is mostly reproducible with high indexing rates (since it also  
> > implies a high rate of segment creation) and merge throttling on (the  
> > default). We are working on publishing a new bugfix release that includes a  
> > fix for this.
> > 
> > [1] [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779)  
> > [https://github.com/elasticsearch/elasticsearch/issues/5779](https://github.com/elasticsearch/elasticsearch/issues/5779)
> > 
> > On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com) wrote:
> > 
> > > Yes, that 's the exact same behavior we're seeing. Bulk indexing  
> > > gradually slows down to about 1/5th of its usual pace.
> > > 
> > > I noted something that might be another clue. During an optimize call:  
> > > every 30th minute there are some I/O operations, according to Marvel, and  
> > > the total segment count, which currently is around 280 000, gets lowered by  
> > > about 120-150 segments, then after another 30 minutes it's lowered again  
> > > with about the same amount of segments. This has been going on ever since  
> > > we first issued the optimize API call after the cluster restart. The  
> > > cluster is not serving any search requests or doing any indexing at the  
> > > time, so can't be that.
> > > 
> > > Robin
> > > 
> > > On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:
> > > 
> > > > I see also exploding numbers in a quick local test on 1.1.0, thousands  
> > > > of segments are created, but I'm not sure if this behavior is expected or  
> > > > not. But that does not look like it should. It slows down bulk indexing  
> > > > significantly.
> > > > 
> > > > Issuing an optimize just stalls...
> > > > 
> > > > curl '0:9200/test/\_optimize?max\_num\_segments=1'
> > > > 
> > > > Does anyone see something similar?
> > > > 
> > > > Stack trace excerpt:
> > > > 
> > > > "elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5  
> > > > os\_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()  
> > > > [0x000000012c140000]  
> > > > java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
> > > > at java.lang.Object.wait(Native Method)  
> > > > at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)
> > > > 
> > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > > > at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite  
> > > > r.java:2262)
> > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > > > at org.elasticsearch.index.engine.internal.InternalEngine.optimize(  
> > > > InternalEngine.java:998)  
> > > > at org.elasticsearch.index.shard.service.InternalIndexShard.opt  
> > > > imize(InternalIndexShard.java:599)  
> > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > imizeAction.shardOperation(TransportOptimizeAction.java:115)  
> > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > imizeAction.shardOperation(TransportOptimizeAction.java:49)  
> > > > at org.elasticsearch.action.support.broadcast.TransportBroadcas  
> > > > tOperationAction$AsyncBroadcastAction$2.run(Tra  
> > > > nsportBroadcastOperationAction.java:226)  
> > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool  
> > > > Executor.java:1142)  
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo  
> > > > lExecutor.java:617)  
> > > > at java.lang.Thread.run(Thread.java:744)
> > > > 
> > > > (node name "Doctor Strange" was not my selection btw)
> > > > 
> > > > Jörg
> > > > 
> > > > On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz [t.b...@trendiction.com](mailto:t.b...@trendiction.com)wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > We also have this issue in 1.0.2. Per index we have over 1000  
> > > > > segments. Optimize doesn't reduce them.
> > > > > 
> > > > > Would love to see a fix for this.
> > > > > 
> > > > > Thanks,  
> > > > > Thibaut
> > > > > 
> > > > > On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com)wrote:
> > > > > 
> > > > > > For reference I'm also on 1.1.0 but I'm not seeing more segments then  
> > > > > > I expect. I see an average of ~28 per shard on an index I write to  
> > > > > > constantly. I don't write all that quickly, \< 50 updates a second.
> > > > > > 
> > > > > > Nik
> > > > > > 
> > > > > > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > 
> > > > > > > I managed to reproduce the issue locally, I'm looking into it.
> > > > > > > 
> > > > > > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > > > > 
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > We are experiencing a related problem with 1.1.0. Segments do not  
> > > > > > > > seem to merge as they should during indexing. The optimize API does  
> > > > > > > > practically nothing in terms of lowering the segments count either. The  
> > > > > > > > problem persists through a cluster restart. The vast amount of segments  
> > > > > > > > seem to be greatly impact the performance of the cluster, in a very  
> > > > > > > > negative way.
> > > > > > > > 
> > > > > > > > We currently have 414 million documents across 3 nodes, each shard  
> > > > > > > > has in average 1200 segments(!).
> > > > > > > > 
> > > > > > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > > > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > > > > > segments per shard back then.
> > > > > > > > 
> > > > > > > > Best Regards,  
> > > > > > > > Robin
> > > > > > > > 
> > > > > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > > > > > 
> > > > > > > > > Thanks for reporting this, the behavior is definitely unexpected.  
> > > > > > > > > I'll test \_optimize on very large numbers of shards to see if I can  
> > > > > > > > > reproduce the issue.
> > > > > > > > > 
> > > > > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<  
> > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > Adrien,
> > > > > > > > > > 
> > > > > > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > > > > > 
> > > > > > > > > > I'm glad I got the index optimized, searches are literally 10-20  
> > > > > > > > > > times faster without 1500/segments per shard to deal with. It's awesome.
> > > > > > > > > > 
> > > > > > > > > > That said, any thoughts on why the index wasn't merging on its  
> > > > > > > > > > own, or why optimize was returning prematurely?
> > > > > > > > > > 
> > > > > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > 
> > > > > > > > > > > I kept the logs up over the last optimize call, and I did see an  
> > > > > > > > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but  
> > > > > > > > > > > I don't think that that caused this exception. The error is essentially as  
> > > > > > > > > > > follows:
> > > > > > > > > > > 
> > > > > > > > > > > netty - Caught exception while handling client http traffic,  
> > > > > > > > > > > closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /  
> > > > > > > > > > > 127.0.0.1:9200]
> > > > > > > > > > > 
> > > > > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown  
> > > > > > > > > > > stream  
> > > > > > > > > > > at Channels.write  
> > > > > > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > at Channels.write  
> > > > > > > > > > > at AbstractChannel.write  
> > > > > > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finis  
> > > > > > > > > > > hHim  
> > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe  
> > > > > > > > > > > ration  
> > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run
> > > > > > > > > > > 
> > > > > > > > > > > Sorry about the crappy stack trace. Still, looks like this  
> > > > > > > > > > > might point to a problem! The exception fired about an hour after I kicked  
> > > > > > > > > > > off the optimize. Any thoughts?
> > > > > > > > > > > 
> > > > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > > 
> > > > > > > > > > > > I did customize my merge policy, although I did so only because  
> > > > > > > > > > > > I was so surprised by the number of segments left over after the load. I'm  
> > > > > > > > > > > > pretty sure the optimize problem was happening before I made this change,  
> > > > > > > > > > > > but either way here are my settings:
> > > > > > > > > > > > 
> > > > > > > > > > > > "index" : {  
> > > > > > > > > > > > "merge" : {  
> > > > > > > > > > > > "policy" : {  
> > > > > > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > > > > > },  
> > > > > > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > > > > > }  
> > > > > > > > > > > > }
> > > > > > > > > > > > 
> > > > > > > > > > > > Not sure whether this set up could be a contributing factor or  
> > > > > > > > > > > > not. Nothing really jumps out at me in the logs. In fact, when i kick off  
> > > > > > > > > > > > the optimize, I don't see any logging at all. Should I?
> > > > > > > > > > > > 
> > > > > > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > > > > > 
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > 
> > > > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand  
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi Elliott,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is  
> > > > > > > > > > > > > not normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > > > > > - Does the issue still reproduce if you restart your cluster?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > reference/current/index-modules-merge.html
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be  
> > > > > > > > > > > > > > significantly impacting performance? Have you guys noticed this behavior  
> > > > > > > > > > > > > > elsewhere?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Adrian,
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I ran the following command:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The  
> > > > > > > > > > > > > > > logs showed "cluster state updated".
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I did have to close my index prior to changing the setting  
> > > > > > > > > > > > > > > and reopen afterward.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I've since began another optimize, but again it doesn't look  
> > > > > > > > > > > > > > > like much is happening. The optimize isn't returning and the total CPU  
> > > > > > > > > > > > > > > usage on every node is holding at about 2% of a single core. I would copy  
> > > > > > > > > > > > > > > a hot\_threads stack trace, but I'm unfortunately on a closed network and  
> > > > > > > > > > > > > > > this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize]  
> > > > > > > > > > > > > > > (doing nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Did you see a message in the logs confirming that the  
> > > > > > > > > > > > > > > > setting has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more  
> > > > > > > > > > > > > > > > > > > times, and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something  
> > > > > > > > > > > > > > > > > > > > was happening in the background, but I'm still seeing over 6500 segments.  
> > > > > > > > > > > > > > > > > > > > Even after setting max\_num\_segments=5. Does this seem right? Queries are  
> > > > > > > > > > > > > > > > > > > > a little faster (350-400ms) but still not great. Bigdesk is still showing  
> > > > > > > > > > > > > > > > > > > > a fair amount of file IO.
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've  
> > > > > > > > > > > > > > > > > > > > > got a 4 node cluster, each with 64G of ram, with 24G allocated to  
> > > > > > > > > > > > > > > > > > > > > Elasticsearch on each. I've batch loaded approximately 86 million  
> > > > > > > > > > > > > > > > > > > > > documents into a single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free  
> > > > > > > > > > > > > > > > > > > > > memory on each server, I would assume that the entire index has been paged  
> > > > > > > > > > > > > > > > > > > > > into memory (I had worried about disk performance previously, as we are  
> > > > > > > > > > > > > > > > > > > > > working in a virtualized environment).
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the  
> > > > > > > > > > > > > > > > > > > > > index is composed of \> 7000 segments. This seemed high to me, but maybe  
> > > > > > > > > > > > > > > > > > > > > it's appropriate. Regardless, I dispatched an optimize command, but I am  
> > > > > > > > > > > > > > > > > > > > > not seeing any progress and the command has not returned. Current merges  
> > > > > > > > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot  
> > > > > > > > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that  
> > > > > > > > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm  
> > > > > > > > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has  
> > > > > > > > > > > > > > > > > > > > > anyone else had similar issues. This is the second time I have tried to  
> > > > > > > > > > > > > > > > > > > > > optimize an index since upgrading. I've gotten the same result both time.
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > > > > from it, send an email to elasticsearc...@googlegroups.c  
> > > > > > > > > > > > > > > > > > > > om.  
> > > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/ms](https://groups.google.com/d/ms)  
> > > > > > > > > > > > > > > > > > > > gid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%  
> > > > > > > > > > > > > > > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send  
> > > > > > > > > > > > > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO  
> > > > > > > > > > > > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)  
> > > > > > > > > > > > > > > > > .  
> > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%](https://groups.google.com/d/msgid/elasticsearch/CAGCt%25)  
> > > > > > > > > > > > > > > > > 2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D  
> > > > > > > > > > > > > > > > > emc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%  
> > > > > > > > > > > > > > > > 2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922)  
> > > > > > > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > --  
> > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > 
> > > > > > > > > > > > --  
> > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d)  
> > > > > > > > > > > > 8-4c2d-a917-dbf53eda03ce%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > .
> > > > > > > > > > 
> > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > Adrien Grand
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit  
> > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e)  
> > > > > > > > c-4574-b989-32727cf3cfe4%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > .
> > > > > > > > 
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > 
> > > > > > > --  
> > > > > > > Adrien Grand
> > > > > > > 
> > > > > > > --  
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr)  
> > > > > > > g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > .
> > > > > > > 
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFG  
> > > > > > uYENwLD6w%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > .
> > > > > > 
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kD  
> > > > > KbB3NMdcA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > .
> > > > > 
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%  
> > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > Adrien Grand
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\
**Post date:** [April 14, 2014, 11:45am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/27 "2014-04-14T11:45:51Z")

</div>

Adrian,

Changing to 'concurrentmergescheduler' did improve my situation, but not  
completely. Optimize would run steadily after this setting was changed,  
but would still time out after approximately an hour. I had to run it  
sequentially about a dozen times to get it to fully optimize over night.

Thanks for digging into this for us!

- Elliott

On Mon, Apr 14, 2014 at 5:43 AM, Adrien Grand \<  
[adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:

> Setting `index.merge.scheduler` to `concurrentmergescheduler`[1] (beware  
> that this is not a live setting so it requires closing the index) should  
> help mitigate the problem.
> 
> Indeed, this fix also implicitely fixes the optimize issue as well: I just  
> ran again my script that reproduced the optimize issue that Elliott  
> described and it doesn't reproduce anymore with the fix.
> 
> [1]  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html#_concurrentmergescheduler)
> 
> On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com) wrote:
> 
> > Great! Is there anything we can do in the meantime in order to decrease  
> > the segment count?
> > 
> > Also, I understand that the issue will solve the amount of segments  
> > created when indexing, but what about optimize? Will it be implicitly fixed  
> > as well?
> > 
> > Thanks!
> > 
> > Robin
> > 
> > On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:
> > 
> > > We found the cause for this issue[1]. There was an issue with the way  
> > > that merges were picked, resulting in too many segments per shard. The  
> > > issue is mostly reproducible with high indexing rates (since it also  
> > > implies a high rate of segment creation) and merge throttling on (the  
> > > default). We are working on publishing a new bugfix release that includes a  
> > > fix for this.
> > > 
> > > [1] [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779)  
> > > [https://github.com/elasticsearch/elasticsearch/issues/5779](https://github.com/elasticsearch/elasticsearch/issues/5779)
> > > 
> > > On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com) wrote:
> > > 
> > > > Yes, that 's the exact same behavior we're seeing. Bulk indexing  
> > > > gradually slows down to about 1/5th of its usual pace.
> > > > 
> > > > I noted something that might be another clue. During an optimize call:  
> > > > every 30th minute there are some I/O operations, according to Marvel, and  
> > > > the total segment count, which currently is around 280 000, gets lowered by  
> > > > about 120-150 segments, then after another 30 minutes it's lowered again  
> > > > with about the same amount of segments. This has been going on ever since  
> > > > we first issued the optimize API call after the cluster restart. The  
> > > > cluster is not serving any search requests or doing any indexing at the  
> > > > time, so can't be that.
> > > > 
> > > > Robin
> > > > 
> > > > On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:
> > > > 
> > > > > I see also exploding numbers in a quick local test on 1.1.0, thousands  
> > > > > of segments are created, but I'm not sure if this behavior is expected or  
> > > > > not. But that does not look like it should. It slows down bulk indexing  
> > > > > significantly.
> > > > > 
> > > > > Issuing an optimize just stalls...
> > > > > 
> > > > > curl '0:9200/test/\_optimize?max\_num\_segments=1'
> > > > > 
> > > > > Does anyone see something similar?
> > > > > 
> > > > > Stack trace excerpt:
> > > > > 
> > > > > "elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5  
> > > > > os\_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()  
> > > > > [0x000000012c140000]  
> > > > > java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
> > > > > at java.lang.Object.wait(Native Method)  
> > > > > at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)
> > > > > 
> > > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > > > > at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite  
> > > > > r.java:2262)
> > > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexWriter)  
> > > > > at org.elasticsearch.index.engine.internal.InternalEngine.optimize(  
> > > > > InternalEngine.java:998)  
> > > > > at org.elasticsearch.index.shard.service.InternalIndexShard.opt  
> > > > > imize(InternalIndexShard.java:599)  
> > > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > > imizeAction.shardOperation(TransportOptimizeAction.java:115)  
> > > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > > imizeAction.shardOperation(TransportOptimizeAction.java:49)  
> > > > > at org.elasticsearch.action.support.broadcast.TransportBroadcas  
> > > > > tOperationAction$AsyncBroadcastAction$2.run(Tra  
> > > > > nsportBroadcastOperationAction.java:226)  
> > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool  
> > > > > Executor.java:1142)  
> > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo  
> > > > > lExecutor.java:617)  
> > > > > at java.lang.Thread.run(Thread.java:744)
> > > > > 
> > > > > (node name "Doctor Strange" was not my selection btw)
> > > > > 
> > > > > Jörg
> > > > > 
> > > > > On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz \<[t.b...@trendiction.com](mailto:t.b...@trendiction.com)
> > > > > 
> > > > > > wrote:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > We also have this issue in 1.0.2. Per index we have over 1000  
> > > > > > segments. Optimize doesn't reduce them.
> > > > > > 
> > > > > > Would love to see a fix for this.
> > > > > > 
> > > > > > Thanks,  
> > > > > > Thibaut
> > > > > > 
> > > > > > On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com)wrote:
> > > > > > 
> > > > > > > For reference I'm also on 1.1.0 but I'm not seeing more segments  
> > > > > > > then I expect. I see an average of ~28 per shard on an index I write to  
> > > > > > > constantly. I don't write all that quickly, \< 50 updates a second.
> > > > > > > 
> > > > > > > Nik
> > > > > > > 
> > > > > > > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > 
> > > > > > > > I managed to reproduce the issue locally, I'm looking into it.
> > > > > > > > 
> > > > > > > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > > > > > 
> > > > > > > > > Hello,
> > > > > > > > > 
> > > > > > > > > We are experiencing a related problem with 1.1.0. Segments do not  
> > > > > > > > > seem to merge as they should during indexing. The optimize API does  
> > > > > > > > > practically nothing in terms of lowering the segments count either. The  
> > > > > > > > > problem persists through a cluster restart. The vast amount of segments  
> > > > > > > > > seem to be greatly impact the performance of the cluster, in a very  
> > > > > > > > > negative way.
> > > > > > > > > 
> > > > > > > > > We currently have 414 million documents across 3 nodes, each shard  
> > > > > > > > > has in average 1200 segments(!).
> > > > > > > > > 
> > > > > > > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > > > > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > > > > > > segments per shard back then.
> > > > > > > > > 
> > > > > > > > > Best Regards,  
> > > > > > > > > Robin
> > > > > > > > > 
> > > > > > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > > > > > > 
> > > > > > > > > > Thanks for reporting this, the behavior is definitely unexpected.  
> > > > > > > > > > I'll test \_optimize on very large numbers of shards to see if I can  
> > > > > > > > > > reproduce the issue.
> > > > > > > > > > 
> > > > > > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<  
> > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > Adrien,
> > > > > > > > > > > 
> > > > > > > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > > > > > > 
> > > > > > > > > > > I'm glad I got the index optimized, searches are literally 10-20  
> > > > > > > > > > > times faster without 1500/segments per shard to deal with. It's awesome.
> > > > > > > > > > > 
> > > > > > > > > > > That said, any thoughts on why the index wasn't merging on its  
> > > > > > > > > > > own, or why optimize was returning prematurely?
> > > > > > > > > > > 
> > > > > > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > > 
> > > > > > > > > > > > I kept the logs up over the last optimize call, and I did see  
> > > > > > > > > > > > an exception. I Ctrl-C'd a curl optimize call before making another one,  
> > > > > > > > > > > > but I don't think that that caused this exception. The error is  
> > > > > > > > > > > > essentially as follows:
> > > > > > > > > > > > 
> > > > > > > > > > > > netty - Caught exception while handling client http traffic,  
> > > > > > > > > > > > closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /  
> > > > > > > > > > > > 127.0.0.1:9200]
> > > > > > > > > > > > 
> > > > > > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > > > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown  
> > > > > > > > > > > > stream  
> > > > > > > > > > > > at Channels.write  
> > > > > > > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > > at Channels.write  
> > > > > > > > > > > > at AbstractChannel.write  
> > > > > > > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finis  
> > > > > > > > > > > > hHim  
> > > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe  
> > > > > > > > > > > > ration  
> > > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run
> > > > > > > > > > > > 
> > > > > > > > > > > > Sorry about the crappy stack trace. Still, looks like this  
> > > > > > > > > > > > might point to a problem! The exception fired about an hour after I kicked  
> > > > > > > > > > > > off the optimize. Any thoughts?
> > > > > > > > > > > > 
> > > > > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I did customize my merge policy, although I did so only  
> > > > > > > > > > > > > because I was so surprised by the number of segments left over after the  
> > > > > > > > > > > > > load. I'm pretty sure the optimize problem was happening before I made  
> > > > > > > > > > > > > this change, but either way here are my settings:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > "index" : {  
> > > > > > > > > > > > > "merge" : {  
> > > > > > > > > > > > > "policy" : {  
> > > > > > > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > > > > > > },  
> > > > > > > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > > > > > > }  
> > > > > > > > > > > > > }
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Not sure whether this set up could be a contributing factor or  
> > > > > > > > > > > > > not. Nothing really jumps out at me in the logs. In fact, when i kick off  
> > > > > > > > > > > > > the optimize, I don't see any logging at all. Should I?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand  
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hi Elliott,
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is  
> > > > > > > > > > > > > > not normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > > > > > > - Does the issue still reproduce if you restart your cluster?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > reference/current/index-modules-merge.html
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard  
> > > > > > > > > > > > > > > be significantly impacting performance? Have you guys noticed this  
> > > > > > > > > > > > > > > behavior elsewhere?
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Adrian,
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > I ran the following command:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The  
> > > > > > > > > > > > > > > > logs showed "cluster state updated".
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > I did have to close my index prior to changing the setting  
> > > > > > > > > > > > > > > > and reopen afterward.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > I've since began another optimize, but again it doesn't  
> > > > > > > > > > > > > > > > look like much is happening. The optimize isn't returning and the total  
> > > > > > > > > > > > > > > > CPU usage on every node is holding at about 2% of a single core. I would  
> > > > > > > > > > > > > > > > copy a hot\_threads stack trace, but I'm unfortunately on a closed network  
> > > > > > > > > > > > > > > > and this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize]  
> > > > > > > > > > > > > > > > (doing nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Did you see a message in the logs confirming that the  
> > > > > > > > > > > > > > > > > setting has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more  
> > > > > > > > > > > > > > > > > > > > times, and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something  
> > > > > > > > > > > > > > > > > > > > > was happening in the background, but I'm still seeing over 6500 segments.  
> > > > > > > > > > > > > > > > > > > > > Even after setting max\_num\_segments=5. Does this seem right? Queries are  
> > > > > > > > > > > > > > > > > > > > > a little faster (350-400ms) but still not great. Bigdesk is still showing  
> > > > > > > > > > > > > > > > > > > > > a fair amount of file IO.
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've  
> > > > > > > > > > > > > > > > > > > > > > got a 4 node cluster, each with 64G of ram, with 24G allocated to  
> > > > > > > > > > > > > > > > > > > > > > Elasticsearch on each. I've batch loaded approximately 86 million  
> > > > > > > > > > > > > > > > > > > > > > documents into a single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free  
> > > > > > > > > > > > > > > > > > > > > > memory on each server, I would assume that the entire index has been paged  
> > > > > > > > > > > > > > > > > > > > > > into memory (I had worried about disk performance previously, as we are  
> > > > > > > > > > > > > > > > > > > > > > working in a virtualized environment).
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that  
> > > > > > > > > > > > > > > > > > > > > > the index is composed of \> 7000 segments. This seemed high to me, but  
> > > > > > > > > > > > > > > > > > > > > > maybe it's appropriate. Regardless, I dispatched an optimize command, but  
> > > > > > > > > > > > > > > > > > > > > > I am not seeing any progress and the command has not returned. Current  
> > > > > > > > > > > > > > > > > > > > > > merges remains at zero, and the segment count is not changing. Checking  
> > > > > > > > > > > > > > > > > > > > > > out hot threads in ElasticHQ, I initially saw an optimize call in the stack  
> > > > > > > > > > > > > > > > > > > > > > that was blocked on a waitForMerge call. This however has disappeared, and  
> > > > > > > > > > > > > > > > > > > > > > I'm seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual?  
> > > > > > > > > > > > > > > > > > > > > > Has anyone else had similar issues. This is the second time I have tried  
> > > > > > > > > > > > > > > > > > > > > > to optimize an index since upgrading. I've gotten the same result both  
> > > > > > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving  
> > > > > > > > > > > > > > > > > > > > > emails from it, send an email to elasticsearc...@  
> > > > > > > > > > > > > > > > > > > > > [googlegroups.com](http://googlegroups.com).  
> > > > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/ms](https://groups.google.com/d/ms)  
> > > > > > > > > > > > > > > > > > > > > gid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%  
> > > > > > > > > > > > > > > > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > a topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send  
> > > > > > > > > > > > > > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO  
> > > > > > > > > > > > > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > > from it, send an email to elasticsearc...@googlegroups.c  
> > > > > > > > > > > > > > > > > > om.  
> > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%](https://groups.google.com/d/msgid/elasticsearch/CAGCt%25)  
> > > > > > > > > > > > > > > > > > 2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D  
> > > > > > > > > > > > > > > > > > emc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an  
> > > > > > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%  
> > > > > > > > > > > > > > > > > 2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922)  
> > > > > > > > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > 
> > > > > > > > > > > > > --  
> > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d)  
> > > > > > > > > > > > > 8-4c2d-a917-dbf53eda03ce%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > .
> > > > > > > > > > > 
> > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > Adrien Grand
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e)  
> > > > > > > > > c-4574-b989-32727cf3cfe4%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > .
> > > > > > > > > 
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > Adrien Grand
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit  
> > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr)  
> > > > > > > > g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > .
> > > > > > > > 
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > 
> > > > > > > --  
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p)  
> > > > > > > FKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > .
> > > > > > > 
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kD  
> > > > > > KbB3NMdcA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > .
> > > > > > 
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%  
> > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .
> > > > 
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > Adrien Grand
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Adrien Grand
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt\_FA\_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [April 14, 2014, 5:37pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/28 "2014-04-14T17:37:05Z")

</div>

With this configuration

> <https://gist.github.com/jprante/10666960>

I could tune the segment merging quite well, on 1.1.0.

Not sure what changed in the default merge settings. I suspect the threads  
of the segment merging could not keep up the segment creation while bulk  
feed under default settings. So I tuned the segments\_per\_tier to 4, the  
merge thread settings, and adjusted some other settings.

After bulk indexing 12,5m docs in 45 minutes, I could run

curl '0:9200/harvard2/\_optimize?max\_num\_segments=1'

as usually - it returned after few minutes (with peaks of 400MB/sec  
read/write on SSD) with a single segment of 10g size.

Helpful is also enabling DEBUG level logging with

index.merge.scheduler: DEBUG  
index.merge.policy: DEBUG

Jörg

On Mon, Apr 14, 2014 at 1:45 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote:

> Adrian,
> 
> Changing to 'concurrentmergescheduler' did improve my situation, but not  
> completely. Optimize would run steadily after this setting was changed,  
> but would still time out after approximately an hour. I had to run it  
> sequentially about a dozen times to get it to fully optimize over night.
> 
> Thanks for digging into this for us!
> 
> - Elliott
> 
> On Mon, Apr 14, 2014 at 5:43 AM, Adrien Grand \<  
> [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:
> 
> > Setting `index.merge.scheduler` to `concurrentmergescheduler`[1] (beware  
> > that this is not a live setting so it requires closing the index) should  
> > help mitigate the problem.
> > 
> > Indeed, this fix also implicitely fixes the optimize issue as well: I  
> > just ran again my script that reproduced the optimize issue that Elliott  
> > described and it doesn't reproduce anymore with the fix.
> > 
> > [1]  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html#_concurrentmergescheduler)
> > 
> > On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com)wrote:
> > 
> > > Great! Is there anything we can do in the meantime in order to decrease  
> > > the segment count?
> > > 
> > > Also, I understand that the issue will solve the amount of segments  
> > > created when indexing, but what about optimize? Will it be implicitly fixed  
> > > as well?
> > > 
> > > Thanks!
> > > 
> > > Robin
> > > 
> > > On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:
> > > 
> > > > We found the cause for this issue[1]. There was an issue with the way  
> > > > that merges were picked, resulting in too many segments per shard. The  
> > > > issue is mostly reproducible with high indexing rates (since it also  
> > > > implies a high rate of segment creation) and merge throttling on (the  
> > > > default). We are working on publishing a new bugfix release that includes a  
> > > > fix for this.
> > > > 
> > > > [1] [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779)  
> > > > [https://github.com/elasticsearch/elasticsearch/issues/5779](https://github.com/elasticsearch/elasticsearch/issues/5779)
> > > > 
> > > > On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > 
> > > > > Yes, that 's the exact same behavior we're seeing. Bulk indexing  
> > > > > gradually slows down to about 1/5th of its usual pace.
> > > > > 
> > > > > I noted something that might be another clue. During an optimize call:  
> > > > > every 30th minute there are some I/O operations, according to Marvel, and  
> > > > > the total segment count, which currently is around 280 000, gets lowered by  
> > > > > about 120-150 segments, then after another 30 minutes it's lowered again  
> > > > > with about the same amount of segments. This has been going on ever since  
> > > > > we first issued the optimize API call after the cluster restart. The  
> > > > > cluster is not serving any search requests or doing any indexing at the  
> > > > > time, so can't be that.
> > > > > 
> > > > > Robin
> > > > > 
> > > > > On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:
> > > > > 
> > > > > > I see also exploding numbers in a quick local test on 1.1.0,  
> > > > > > thousands of segments are created, but I'm not sure if this behavior is  
> > > > > > expected or not. But that does not look like it should. It slows down bulk  
> > > > > > indexing significantly.
> > > > > > 
> > > > > > Issuing an optimize just stalls...
> > > > > > 
> > > > > > curl '0:9200/test/\_optimize?max\_num\_segments=1'
> > > > > > 
> > > > > > Does anyone see something similar?
> > > > > > 
> > > > > > Stack trace excerpt:
> > > > > > 
> > > > > > "elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5  
> > > > > > os\_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()  
> > > > > > [0x000000012c140000]  
> > > > > > java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
> > > > > > at java.lang.Object.wait(Native Method)  
> > > > > > at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)
> > > > > > 
> > > > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexW  
> > > > > > riter)  
> > > > > > at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite  
> > > > > > r.java:2262)
> > > > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexW  
> > > > > > riter)  
> > > > > > at org.elasticsearch.index.engine.internal.InternalEngine.optimize(  
> > > > > > InternalEngine.java:998)  
> > > > > > at org.elasticsearch.index.shard.service.InternalIndexShard.opt  
> > > > > > imize(InternalIndexShard.java:599)  
> > > > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > > > imizeAction.shardOperation(TransportOptimizeAction.java:115)  
> > > > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > > > imizeAction.shardOperation(TransportOptimizeAction.java:49)  
> > > > > > at org.elasticsearch.action.support.broadcast.TransportBroadcas  
> > > > > > tOperationAction$AsyncBroadcastAction$2.run(Tra  
> > > > > > nsportBroadcastOperationAction.java:226)  
> > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool  
> > > > > > Executor.java:1142)  
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo  
> > > > > > lExecutor.java:617)  
> > > > > > at java.lang.Thread.run(Thread.java:744)
> > > > > > 
> > > > > > (node name "Doctor Strange" was not my selection btw)
> > > > > > 
> > > > > > Jörg
> > > > > > 
> > > > > > On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz \<  
> > > > > > [t.b...@trendiction.com](mailto:t.b...@trendiction.com)\> wrote:
> > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > We also have this issue in 1.0.2. Per index we have over 1000  
> > > > > > > segments. Optimize doesn't reduce them.
> > > > > > > 
> > > > > > > Would love to see a fix for this.
> > > > > > > 
> > > > > > > Thanks,  
> > > > > > > Thibaut
> > > > > > > 
> > > > > > > On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com)wrote:
> > > > > > > 
> > > > > > > > For reference I'm also on 1.1.0 but I'm not seeing more segments  
> > > > > > > > then I expect. I see an average of ~28 per shard on an index I write to  
> > > > > > > > constantly. I don't write all that quickly, \< 50 updates a second.
> > > > > > > > 
> > > > > > > > Nik
> > > > > > > > 
> > > > > > > > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > 
> > > > > > > > > I managed to reproduce the issue locally, I'm looking into it.
> > > > > > > > > 
> > > > > > > > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > > > > > > 
> > > > > > > > > > Hello,
> > > > > > > > > > 
> > > > > > > > > > We are experiencing a related problem with 1.1.0. Segments do not  
> > > > > > > > > > seem to merge as they should during indexing. The optimize API does  
> > > > > > > > > > practically nothing in terms of lowering the segments count either. The  
> > > > > > > > > > problem persists through a cluster restart. The vast amount of segments  
> > > > > > > > > > seem to be greatly impact the performance of the cluster, in a very  
> > > > > > > > > > negative way.
> > > > > > > > > > 
> > > > > > > > > > We currently have 414 million documents across 3 nodes, each  
> > > > > > > > > > shard has in average 1200 segments(!).
> > > > > > > > > > 
> > > > > > > > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > > > > > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > > > > > > > segments per shard back then.
> > > > > > > > > > 
> > > > > > > > > > Best Regards,  
> > > > > > > > > > Robin
> > > > > > > > > > 
> > > > > > > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > > > > > > > 
> > > > > > > > > > > Thanks for reporting this, the behavior is definitely  
> > > > > > > > > > > unexpected. I'll test \_optimize on very large numbers of shards to see if I  
> > > > > > > > > > > can reproduce the issue.
> > > > > > > > > > > 
> > > > > > > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<  
> > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Adrien,
> > > > > > > > > > > > 
> > > > > > > > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > > > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > > > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > > > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > > > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > > > > > > > 
> > > > > > > > > > > > I'm glad I got the index optimized, searches are literally  
> > > > > > > > > > > > 10-20 times faster without 1500/segments per shard to deal with. It's  
> > > > > > > > > > > > awesome.
> > > > > > > > > > > > 
> > > > > > > > > > > > That said, any thoughts on why the index wasn't merging on its  
> > > > > > > > > > > > own, or why optimize was returning prematurely?
> > > > > > > > > > > > 
> > > > > > > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I kept the logs up over the last optimize call, and I did see  
> > > > > > > > > > > > > an exception. I Ctrl-C'd a curl optimize call before making another one,  
> > > > > > > > > > > > > but I don't think that that caused this exception. The error is  
> > > > > > > > > > > > > essentially as follows:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > netty - Caught exception while handling client http traffic,  
> > > > > > > > > > > > > closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /  
> > > > > > > > > > > > > 127.0.0.1:9200]
> > > > > > > > > > > > > 
> > > > > > > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker.  
> > > > > > > > > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433)  
> > > > > > > > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.  
> > > > > > > > > > > > > sendDownstream  
> > > > > > > > > > > > > at Channels.write  
> > > > > > > > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > > > at Channels.write  
> > > > > > > > > > > > > at AbstractChannel.write  
> > > > > > > > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.  
> > > > > > > > > > > > > finishHim  
> > > > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.  
> > > > > > > > > > > > > onOperation  
> > > > > > > > > > > > > at TransportBroadcastOperationAct  
> > > > > > > > > > > > > ion$AsyncBroadcastAction$2.run
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Sorry about the crappy stack trace. Still, looks like this  
> > > > > > > > > > > > > might point to a problem! The exception fired about an hour after I kicked  
> > > > > > > > > > > > > off the optimize. Any thoughts?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott  
> > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I did customize my merge policy, although I did so only  
> > > > > > > > > > > > > > because I was so surprised by the number of segments left over after the  
> > > > > > > > > > > > > > load. I'm pretty sure the optimize problem was happening before I made  
> > > > > > > > > > > > > > this change, but either way here are my settings:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > "index" : {  
> > > > > > > > > > > > > > "merge" : {  
> > > > > > > > > > > > > > "policy" : {  
> > > > > > > > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > > > > > > > },  
> > > > > > > > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > > > > > > > }  
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Not sure whether this set up could be a contributing factor  
> > > > > > > > > > > > > > or not. Nothing really jumps out at me in the logs. In fact, when i kick  
> > > > > > > > > > > > > > off the optimize, I don't see any logging at all. Should I?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand  
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Hi Elliott,
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is  
> > > > > > > > > > > > > > > not normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > > > > > > > - Does the issue still reproduce if you restart your  
> > > > > > > > > > > > > > > cluster?
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > > reference/current/index-modules-merge.html
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard  
> > > > > > > > > > > > > > > > be significantly impacting performance? Have you guys noticed this  
> > > > > > > > > > > > > > > > behavior elsewhere?
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw  
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Adrian,
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > I ran the following command:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The  
> > > > > > > > > > > > > > > > > logs showed "cluster state updated".
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > I did have to close my index prior to changing the setting  
> > > > > > > > > > > > > > > > > and reopen afterward.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > I've since began another optimize, but again it doesn't  
> > > > > > > > > > > > > > > > > look like much is happening. The optimize isn't returning and the total  
> > > > > > > > > > > > > > > > > CPU usage on every node is holding at about 2% of a single core. I would  
> > > > > > > > > > > > > > > > > copy a hot\_threads stack trace, but I'm unfortunately on a closed network  
> > > > > > > > > > > > > > > > > and this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize]  
> > > > > > > > > > > > > > > > > (doing nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Did you see a message in the logs confirming that the  
> > > > > > > > > > > > > > > > > > setting has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more  
> > > > > > > > > > > > > > > > > > > > > times, and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose  
> > > > > > > > > > > > > > > > > > > > > > something was happening in the background, but I'm still seeing over 6500  
> > > > > > > > > > > > > > > > > > > > > > segments. Even after setting max\_num\_segments=5. Does this seem right?  
> > > > > > > > > > > > > > > > > > > > > > Queries are a little faster (350-400ms) but still not great. Bigdesk is  
> > > > > > > > > > > > > > > > > > > > > > still showing a fair amount of file IO.
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've  
> > > > > > > > > > > > > > > > > > > > > > > got a 4 node cluster, each with 64G of ram, with 24G allocated to  
> > > > > > > > > > > > > > > > > > > > > > > Elasticsearch on each. I've batch loaded approximately 86 million  
> > > > > > > > > > > > > > > > > > > > > > > documents into a single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free  
> > > > > > > > > > > > > > > > > > > > > > > memory on each server, I would assume that the entire index has been paged  
> > > > > > > > > > > > > > > > > > > > > > > into memory (I had worried about disk performance previously, as we are  
> > > > > > > > > > > > > > > > > > > > > > > working in a virtualized environment).
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that  
> > > > > > > > > > > > > > > > > > > > > > > the index is composed of \> 7000 segments. This seemed high to me, but  
> > > > > > > > > > > > > > > > > > > > > > > maybe it's appropriate. Regardless, I dispatched an optimize command, but  
> > > > > > > > > > > > > > > > > > > > > > > I am not seeing any progress and the command has not returned. Current  
> > > > > > > > > > > > > > > > > > > > > > > merges remains at zero, and the segment count is not changing. Checking  
> > > > > > > > > > > > > > > > > > > > > > > out hot threads in ElasticHQ, I initially saw an optimize call in the stack  
> > > > > > > > > > > > > > > > > > > > > > > that was blocked on a waitForMerge call. This however has disappeared, and  
> > > > > > > > > > > > > > > > > > > > > > > I'm seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual?  
> > > > > > > > > > > > > > > > > > > > > > > Has anyone else had similar issues. This is the second time I have tried  
> > > > > > > > > > > > > > > > > > > > > > > to optimize an index since upgrading. I've gotten the same result both  
> > > > > > > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > > > > You received this message because you are subscribed  
> > > > > > > > > > > > > > > > > > > > > > to the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving  
> > > > > > > > > > > > > > > > > > > > > > emails from it, send an email to elasticsearc...@  
> > > > > > > > > > > > > > > > > > > > > > [googlegroups.com](http://googlegroups.com).  
> > > > > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/ms](https://groups.google.com/d/ms)  
> > > > > > > > > > > > > > > > > > > > > > gid/elasticsearch/5391291f-5c5  
> > > > > > > > > > > > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > > a topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send  
> > > > > > > > > > > > > > > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO  
> > > > > > > > > > > > > > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > > > from it, send an email to elasticsearc...@googlegroups.c  
> > > > > > > > > > > > > > > > > > > om.  
> > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%](https://groups.google.com/d/msgid/elasticsearch/CAGCt%25)  
> > > > > > > > > > > > > > > > > > > 2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D  
> > > > > > > > > > > > > > > > > > > emc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > You received this message because you are subscribed to a  
> > > > > > > > > > > > > > > > > > topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send  
> > > > > > > > > > > > > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > > CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%  
> > > > > > > > > > > > > > > > > > 2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-](https://groups.google.com/d/msgid/elasticsearch/8742280e-)  
> > > > > > > > > > > > > > > > > 922f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d)  
> > > > > > > > > > > > > > 8-4c2d-a917-dbf53eda03ce%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > .
> > > > > > > > > > > > 
> > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > 
> > > > > > > > > > > --  
> > > > > > > > > > > Adrien Grand
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e)  
> > > > > > > > > > c-4574-b989-32727cf3cfe4%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > .
> > > > > > > > > > 
> > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > Adrien Grand
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr)  
> > > > > > > > > g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > .
> > > > > > > > > 
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit  
> > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p)  
> > > > > > > > FKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > .
> > > > > > > > 
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > 
> > > > > > > --  
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKY](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKY)  
> > > > > > > tkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > .
> > > > > > > 
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%  
> > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > .
> > > > > 
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > Adrien Grand
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > Adrien Grand
> > 
> > --  
> > You received this message because you are subscribed to a topic in the  
> > Google Groups "elasticsearch" group.  
> > To unsubscribe from this topic, visit  
> > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe).  
> > To unsubscribe from this group and all its topics, send an email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt\_FA\_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt\_FA\_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH5ZfojVsoNEdUbKnxhSFsh\_teS5hKxbGdHgg4%3D0XT8zQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH5ZfojVsoNEdUbKnxhSFsh_teS5hKxbGdHgg4%3D0XT8zQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\
**Post date:** [May 13, 2014, 7:45pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/29 "2014-05-13T19:45:01Z")

</div>

Jorg,

I've just had a chance to get back to this and your configuration is  
working great. Merges are being dispatched as expected, and the total  
segment count during loading is staying in the hundreds. Nice stuff!

On Monday, April 14, 2014 1:37:05 PM UTC-4, Jörg Prante wrote:

> With this configuration
> 
> [Elasticsearch configuration for high sustainable bulk feed · GitHub](https://gist.github.com/jprante/10666960)
> 
> I could tune the segment merging quite well, on 1.1.0.
> 
> Not sure what changed in the default merge settings. I suspect the threads  
> of the segment merging could not keep up the segment creation while bulk  
> feed under default settings. So I tuned the segments\_per\_tier to 4, the  
> merge thread settings, and adjusted some other settings.
> 
> After bulk indexing 12,5m docs in 45 minutes, I could run
> 
> curl '0:9200/harvard2/\_optimize?max\_num\_segments=1'
> 
> as usually - it returned after few minutes (with peaks of 400MB/sec  
> read/write on SSD) with a single segment of 10g size.
> 
> Helpful is also enabling DEBUG level logging with
> 
> index.merge.scheduler: DEBUG  
> index.merge.policy: DEBUG
> 
> Jörg
> 
> On Mon, Apr 14, 2014 at 1:45 PM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > Adrian,
> > 
> > Changing to 'concurrentmergescheduler' did improve my situation, but not  
> > completely. Optimize would run steadily after this setting was changed,  
> > but would still time out after approximately an hour. I had to run it  
> > sequentially about a dozen times to get it to fully optimize over night.
> > 
> > Thanks for digging into this for us!
> > 
> > - Elliott
> > 
> > On Mon, Apr 14, 2014 at 5:43 AM, Adrien Grand \<  
> > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com) \<javascript:\>\> wrote:
> > 
> > > Setting `index.merge.scheduler` to `concurrentmergescheduler`[1] (beware  
> > > that this is not a live setting so it requires closing the index) should  
> > > help mitigate the problem.
> > > 
> > > Indeed, this fix also implicitely fixes the optimize issue as well: I  
> > > just ran again my script that reproduced the optimize issue that Elliott  
> > > described and it doesn't reproduce anymore with the fix.
> > > 
> > > [1]  
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html#_concurrentmergescheduler)
> > > 
> > > On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin \<[walr...@gmail.com](mailto:walr...@gmail.com)\<javascript:\>
> > > 
> > > > wrote:
> > > 
> > > > Great! Is there anything we can do in the meantime in order to decrease  
> > > > the segment count?
> > > > 
> > > > Also, I understand that the issue will solve the amount of segments  
> > > > created when indexing, but what about optimize? Will it be implicitly fixed  
> > > > as well?
> > > > 
> > > > Thanks!
> > > > 
> > > > Robin
> > > > 
> > > > On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:
> > > > 
> > > > > We found the cause for this issue[1]. There was an issue with the way  
> > > > > that merges were picked, resulting in too many segments per shard. The  
> > > > > issue is mostly reproducible with high indexing rates (since it also  
> > > > > implies a high rate of segment creation) and merge throttling on (the  
> > > > > default). We are working on publishing a new bugfix release that includes a  
> > > > > fix for this.
> > > > > 
> > > > > [1] [Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5779)  
> > > > > [https://github.com/elasticsearch/elasticsearch/issues/5779](https://github.com/elasticsearch/elasticsearch/issues/5779)
> > > > > 
> > > > > On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > > 
> > > > > > Yes, that 's the exact same behavior we're seeing. Bulk indexing  
> > > > > > gradually slows down to about 1/5th of its usual pace.
> > > > > > 
> > > > > > I noted something that might be another clue. During an optimize  
> > > > > > call: every 30th minute there are some I/O operations, according to Marvel,  
> > > > > > and the total segment count, which currently is around 280 000, gets  
> > > > > > lowered by about 120-150 segments, then after another 30 minutes it's  
> > > > > > lowered again with about the same amount of segments. This has been going  
> > > > > > on ever since we first issued the optimize API call after the cluster  
> > > > > > restart. The cluster is not serving any search requests or doing any  
> > > > > > indexing at the time, so can't be that.
> > > > > > 
> > > > > > Robin
> > > > > > 
> > > > > > On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:
> > > > > > 
> > > > > > > I see also exploding numbers in a quick local test on 1.1.0,  
> > > > > > > thousands of segments are created, but I'm not sure if this behavior is  
> > > > > > > expected or not. But that does not look like it should. It slows down bulk  
> > > > > > > indexing significantly.
> > > > > > > 
> > > > > > > Issuing an optimize just stalls...
> > > > > > > 
> > > > > > > curl '0:9200/test/\_optimize?max\_num\_segments=1'
> > > > > > > 
> > > > > > > Does anyone see something similar?
> > > > > > > 
> > > > > > > Stack trace excerpt:
> > > > > > > 
> > > > > > > "elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5  
> > > > > > > os\_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()  
> > > > > > > [0x000000012c140000]  
> > > > > > > java.lang.Thread.State: TIMED\_WAITING (on object monitor)  
> > > > > > > at java.lang.Object.wait(Native Method)  
> > > > > > > at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)
> > > > > > > 
> > > > > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexW  
> > > > > > > riter)  
> > > > > > > at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite  
> > > > > > > r.java:2262)
> > > > > > > - locked \<0x00000006cee0fad0\> (a org.apache.lucene.index.IndexW  
> > > > > > > riter)  
> > > > > > > at org.elasticsearch.index.engine.internal.InternalEngine.optimize(  
> > > > > > > InternalEngine.java:998)  
> > > > > > > at org.elasticsearch.index.shard.service.InternalIndexShard.opt  
> > > > > > > imize(InternalIndexShard.java:599)  
> > > > > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > > > > imizeAction.shardOperation(TransportOptimizeAction.java:115)  
> > > > > > > at org.elasticsearch.action.admin.indices.optimize.TransportOpt  
> > > > > > > imizeAction.shardOperation(TransportOptimizeAction.java:49)  
> > > > > > > at org.elasticsearch.action.support.broadcast.TransportBroadcas  
> > > > > > > tOperationAction$AsyncBroadcastAction$2.run(Tra  
> > > > > > > nsportBroadcastOperationAction.java:226)  
> > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool  
> > > > > > > Executor.java:1142)  
> > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo  
> > > > > > > lExecutor.java:617)  
> > > > > > > at java.lang.Thread.run(Thread.java:744)
> > > > > > > 
> > > > > > > (node name "Doctor Strange" was not my selection btw)
> > > > > > > 
> > > > > > > Jörg
> > > > > > > 
> > > > > > > On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz \<  
> > > > > > > [t.b...@trendiction.com](mailto:t.b...@trendiction.com)\> wrote:
> > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > We also have this issue in 1.0.2. Per index we have over 1000  
> > > > > > > > segments. Optimize doesn't reduce them.
> > > > > > > > 
> > > > > > > > Would love to see a fix for this.
> > > > > > > > 
> > > > > > > > Thanks,  
> > > > > > > > Thibaut
> > > > > > > > 
> > > > > > > > On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com)wrote:
> > > > > > > > 
> > > > > > > > > For reference I'm also on 1.1.0 but I'm not seeing more segments  
> > > > > > > > > then I expect. I see an average of ~28 per shard on an index I write to  
> > > > > > > > > constantly. I don't write all that quickly, \< 50 updates a second.
> > > > > > > > > 
> > > > > > > > > Nik
> > > > > > > > > 
> > > > > > > > > On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \<  
> > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > I managed to reproduce the issue locally, I'm looking into it.
> > > > > > > > > > 
> > > > > > > > > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walr...@gmail.com](mailto:walr...@gmail.com)wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hello,
> > > > > > > > > > > 
> > > > > > > > > > > We are experiencing a related problem with 1.1.0. Segments do  
> > > > > > > > > > > not seem to merge as they should during indexing. The optimize API does  
> > > > > > > > > > > practically nothing in terms of lowering the segments count either. The  
> > > > > > > > > > > problem persists through a cluster restart. The vast amount of segments  
> > > > > > > > > > > seem to be greatly impact the performance of the cluster, in a very  
> > > > > > > > > > > negative way.
> > > > > > > > > > > 
> > > > > > > > > > > We currently have 414 million documents across 3 nodes, each  
> > > > > > > > > > > shard has in average 1200 segments(!).
> > > > > > > > > > > 
> > > > > > > > > > > With 1.0.1 we had even more documents, ~650 million, without any  
> > > > > > > > > > > segment problems. Looking in Marvel we were hovering at around 30-40  
> > > > > > > > > > > segments per shard back then.
> > > > > > > > > > > 
> > > > > > > > > > > Best Regards,  
> > > > > > > > > > > Robin
> > > > > > > > > > > 
> > > > > > > > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Thanks for reporting this, the behavior is definitely  
> > > > > > > > > > > > unexpected. I'll test \_optimize on very large numbers of shards to see if I  
> > > > > > > > > > > > can reproduce the issue.
> > > > > > > > > > > > 
> > > > > > > > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Adrien,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Just an FYI, after resetting the cluster, things seem to have  
> > > > > > > > > > > > > improved. Optimize calls now lead to CPU/IO activity over their duration.  
> > > > > > > > > > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as  
> > > > > > > > > > > > > each call would only reduce the segment count by about 600-700. I ran 10  
> > > > > > > > > > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)!
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I'm glad I got the index optimized, searches are literally  
> > > > > > > > > > > > > 10-20 times faster without 1500/segments per shard to deal with. It's  
> > > > > > > > > > > > > awesome.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > That said, any thoughts on why the index wasn't merging on its  
> > > > > > > > > > > > > own, or why optimize was returning prematurely?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott  
> > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I kept the logs up over the last optimize call, and I did see  
> > > > > > > > > > > > > > an exception. I Ctrl-C'd a curl optimize call before making another one,  
> > > > > > > > > > > > > > but I don't think that that caused this exception. The error is  
> > > > > > > > > > > > > > essentially as follows:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > netty - Caught exception while handling client http traffic,  
> > > > > > > > > > > > > > closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /  
> > > > > > > > > > > > > > 127.0.0.1:9200]
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > java.nio.channels.ClosedChannelException at  
> > > > > > > > > > > > > > AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:  
> > > > > > > > > > > > > > 433)  
> > > > > > > > > > > > > > at AbstractNioWorker.writeFromUserCode  
> > > > > > > > > > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket  
> > > > > > > > > > > > > > at NioServerSocketPipelineSink.eventSunk  
> > > > > > > > > > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.  
> > > > > > > > > > > > > > sendDownstream  
> > > > > > > > > > > > > > at Channels.write  
> > > > > > > > > > > > > > at OneToOneEncoder.doEncode  
> > > > > > > > > > > > > > at OneToOneEncoder.handleDownstream  
> > > > > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > > > > at DefaultChannelPipeline.sendDownstream  
> > > > > > > > > > > > > > at Channels.write  
> > > > > > > > > > > > > > at AbstractChannel.write  
> > > > > > > > > > > > > > at NettyHttpChannel.sendResponse  
> > > > > > > > > > > > > > at RestOptimizeAction$1.onResponse(95)  
> > > > > > > > > > > > > > at RestOptimizeAction$1.onResponse(85)  
> > > > > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.  
> > > > > > > > > > > > > > finishHim  
> > > > > > > > > > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.  
> > > > > > > > > > > > > > onOperation  
> > > > > > > > > > > > > > at TransportBroadcastOperationAct  
> > > > > > > > > > > > > > ion$AsyncBroadcastAction$2.run
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Sorry about the crappy stack trace. Still, looks like this  
> > > > > > > > > > > > > > might point to a problem! The exception fired about an hour after I kicked  
> > > > > > > > > > > > > > off the optimize. Any thoughts?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott  
> > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Hi Adrien,
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I did customize my merge policy, although I did so only  
> > > > > > > > > > > > > > > because I was so surprised by the number of segments left over after the  
> > > > > > > > > > > > > > > load. I'm pretty sure the optimize problem was happening before I made  
> > > > > > > > > > > > > > > this change, but either way here are my settings:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > "index" : {  
> > > > > > > > > > > > > > > "merge" : {  
> > > > > > > > > > > > > > > "policy" : {  
> > > > > > > > > > > > > > > "max\_merged\_segment" : "20gb",  
> > > > > > > > > > > > > > > "segments\_per\_tier" : 5,  
> > > > > > > > > > > > > > > "floor\_segment" : "10mb"  
> > > > > > > > > > > > > > > },  
> > > > > > > > > > > > > > > "scheduler" : "concurrentmergescheduler"  
> > > > > > > > > > > > > > > }  
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Not sure whether this set up could be a contributing factor  
> > > > > > > > > > > > > > > or not. Nothing really jumps out at me in the logs. In fact, when i kick  
> > > > > > > > > > > > > > > off the optimize, I don't see any logging at all. Should I?
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I'm running the following command: curl -XPOST  
> > > > > > > > > > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize)
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand  
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Hi Elliott,
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > 1500 segments per shard is certainly way too much, and it  
> > > > > > > > > > > > > > > > is not normal that optimize doesn't manage to reduce the number of segments.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > - Is there anything suspicious in the logs?
> > > > > > > > > > > > > > > > - Have you customized the merge policy or scheduler?[1]
> > > > > > > > > > > > > > > > - Does the issue still reproduce if you restart your  
> > > > > > > > > > > > > > > > cluster?
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > > > reference/current/index-modules-merge.html
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard  
> > > > > > > > > > > > > > > > > be significantly impacting performance? Have you guys noticed this  
> > > > > > > > > > > > > > > > > behavior elsewhere?
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > Adrian,
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > I ran the following command:
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d  
> > > > > > > > > > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}'
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The  
> > > > > > > > > > > > > > > > > > logs showed "cluster state updated".
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > I did have to close my index prior to changing the  
> > > > > > > > > > > > > > > > > > setting and reopen afterward.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > I've since began another optimize, but again it doesn't  
> > > > > > > > > > > > > > > > > > look like much is happening. The optimize isn't returning and the total  
> > > > > > > > > > > > > > > > > > CPU usage on every node is holding at about 2% of a single core. I would  
> > > > > > > > > > > > > > > > > > copy a hot\_threads stack trace, but I'm unfortunately on a closed network  
> > > > > > > > > > > > > > > > > > and this isn't possible. I can tell you that refreshes of hot\_threads show  
> > > > > > > > > > > > > > > > > > vary little happening. The occasional [merge] thread (always in a  
> > > > > > > > > > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize]  
> > > > > > > > > > > > > > > > > > (doing nothing on a waitForMerge() call) thread shows up, but it's always  
> > > > > > > > > > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any  
> > > > > > > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<  
> > > > > > > > > > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote:
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > Did you see a message in the logs confirming that the  
> > > > > > > > > > > > > > > > > > > setting has been updated? It would be interesting to see the output of hot  
> > > > > > > > > > > > > > > > > > > threads[1] to see what your node is doing.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > reference/current/cluster-nodes-hot-threads.html
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time.
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \<  
> > > > > > > > > > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote:
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize?
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<  
> > > > > > > > > > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote:
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more  
> > > > > > > > > > > > > > > > > > > > > > times, and the number of segments falls each time, but I'm still over 1000  
> > > > > > > > > > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar?
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose  
> > > > > > > > > > > > > > > > > > > > > > > something was happening in the background, but I'm still seeing over 6500  
> > > > > > > > > > > > > > > > > > > > > > > segments. Even after setting max\_num\_segments=5. Does this seem right?  
> > > > > > > > > > > > > > > > > > > > > > > Queries are a little faster (350-400ms) but still not great. Bigdesk is  
> > > > > > > > > > > > > > > > > > > > > > > still showing a fair amount of file IO.
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott  
> > > > > > > > > > > > > > > > > > > > > > > Bradshaw wrote:
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0.  
> > > > > > > > > > > > > > > > > > > > > > > > I've got a 4 node cluster, each with 64G of ram, with 24G allocated to  
> > > > > > > > > > > > > > > > > > > > > > > > Elasticsearch on each. I've batch loaded approximately 86 million  
> > > > > > > > > > > > > > > > > > > > > > > > documents into a single index (4 shards) and have started benchmarking  
> > > > > > > > > > > > > > > > > > > > > > > > cross\_field/multi\_match queries on them. The index has one replica and  
> > > > > > > > > > > > > > > > > > > > > > > > takes up a total of 111G. I've run several batches of warming queries, but  
> > > > > > > > > > > > > > > > > > > > > > > > queries are not as fast as I had hoped, approximately 400-500ms each.  
> > > > > > > > > > > > > > > > > > > > > > > > Given that \*top \*(on Centos) shows 5-8 GB of free  
> > > > > > > > > > > > > > > > > > > > > > > > memory on each server, I would assume that the entire index has been paged  
> > > > > > > > > > > > > > > > > > > > > > > > into memory (I had worried about disk performance previously, as we are  
> > > > > > > > > > > > > > > > > > > > > > > > working in a virtualized environment).
> > > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that  
> > > > > > > > > > > > > > > > > > > > > > > > the index is composed of \> 7000 segments. This seemed high to me, but  
> > > > > > > > > > > > > > > > > > > > > > > > maybe it's appropriate. Regardless, I dispatched an optimize command, but  
> > > > > > > > > > > > > > > > > > > > > > > > I am not seeing any progress and the command has not returned. Current  
> > > > > > > > > > > > > > > > > > > > > > > > merges remains at zero, and the segment count is not changing. Checking  
> > > > > > > > > > > > > > > > > > > > > > > > out hot threads in ElasticHQ, I initially saw an optimize call in the stack  
> > > > > > > > > > > > > > > > > > > > > > > > that was blocked on a waitForMerge call. This however has disappeared, and  
> > > > > > > > > > > > > > > > > > > > > > > > I'm seeing no evidence that the optimize is occuring.
> > > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual?  
> > > > > > > > > > > > > > > > > > > > > > > > Has anyone else had similar issues. This is the second time I have tried  
> > > > > > > > > > > > > > > > > > > > > > > > to optimize an index since upgrading. I've gotten the same result both  
> > > > > > > > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips!
> > > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > > - Elliott
> > > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > > > > > You received this message because you are subscribed  
> > > > > > > > > > > > > > > > > > > > > > > to the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving  
> > > > > > > > > > > > > > > > > > > > > > > emails from it, send an email to elasticsearc...@  
> > > > > > > > > > > > > > > > > > > > > > > [googlegroups.com](http://googlegroups.com).  
> > > > > > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/ms](https://groups.google.com/d/ms)  
> > > > > > > > > > > > > > > > > > > > > > > gid/elasticsearch/5391291f-5c5  
> > > > > > > > > > > > > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > > > > > > > > > > > > > > > > > optout.
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > > > You received this message because you are subscribed  
> > > > > > > > > > > > > > > > > > > > > to a topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics,  
> > > > > > > > > > > > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO  
> > > > > > > > > > > > > > > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving  
> > > > > > > > > > > > > > > > > > > > emails from it, send an email to elasticsearc...@  
> > > > > > > > > > > > > > > > > > > > [googlegroups.com](http://googlegroups.com).  
> > > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%](https://groups.google.com/d/msgid/elasticsearch/CAGCt%25)  
> > > > > > > > > > > > > > > > > > > > 2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D  
> > > > > > > > > > > > > > > > > > > > emc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > > a topic in the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > > To unsubscribe from this topic, visit  
> > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/](https://groups.google.com/d/topic/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > kqTRRADQBwc/unsubscribe.  
> > > > > > > > > > > > > > > > > > > To unsubscribe from this group and all its topics, send  
> > > > > > > > > > > > > > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/)  
> > > > > > > > > > > > > > > > > > > CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%  
> > > > > > > > > > > > > > > > > > > 2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/op](https://groups.google.com/d/op)  
> > > > > > > > > > > > > > > > > > > tout.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > > > You received this message because you are subscribed to  
> > > > > > > > > > > > > > > > > > the Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails  
> > > > > > > > > > > > > > > > > > from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)  
> > > > > > > > > > > > > > > > > > .  
> > > > > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-](https://groups.google.com/d/msgid/elasticsearch/8742280e-)  
> > > > > > > > > > > > > > > > > > 922f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout)  
> > > > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > --  
> > > > > > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d)  
> > > > > > > > > > > > > > > 8-4c2d-a917-dbf53eda03ce%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > > > > > .
> > > > > > > > > > > > > 
> > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > > > 
> > > > > > > > > > > > --  
> > > > > > > > > > > > Adrien Grand
> > > > > > > > > > > 
> > > > > > > > > > > --  
> > > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > > To unsubscribe from this group and stop receiving emails from  
> > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e)  
> > > > > > > > > > > c-4574-b989-32727cf3cfe4%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > > .
> > > > > > > > > > > 
> > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > Adrien Grand
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > You received this message because you are subscribed to the  
> > > > > > > > > > Google Groups "elasticsearch" group.  
> > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr)  
> > > > > > > > > > g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > > .
> > > > > > > > > > 
> > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit  
> > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p)  
> > > > > > > > > FKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > > .
> > > > > > > > > 
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > > Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit  
> > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKY](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKY)  
> > > > > > > > tkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAE\_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > > > > > .
> > > > > > > > 
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > 
> > > > > > > --  
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > > > > > msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%  
> > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > .
> > > > > > 
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > Adrien Grand
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .
> > > > 
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > Adrien Grand
> > > 
> > > --  
> > > You received this message because you are subscribed to a topic in the  
> > > Google Groups "elasticsearch" group.  
> > > To unsubscribe from this topic, visit  
> > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe).  
> > > To unsubscribe from this group and all its topics, send an email to  
> > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt\_FA\_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt\_FA\_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ba4373b4-c895-4339-845d-488907d1c824%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ba4373b4-c895-4339-845d-488907d1c824%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:29am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/30 "2017-07-06T01:29:43Z")

</div>



[Previous page](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790.md?page=1)