# Elasticsearch 1.1.0 - Optimize broken? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 3, 2014, 12:47pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790 "2014-04-03T12:47:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 3, 2014, 12:47pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/1 "2014-04-03T12:47:32Z")
Hi All, I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch on each. I've batch loaded approximately 86 million documents into a single index (4 shards) and have started benchmarking cross\_field/multi\_match queries on them. The index has one replica and takes up a total of 111G. I've run several batches of warming queries, but queries are not as fast as I had hoped, approximately 400-500ms each. Given that \*top \*(on Centos) shows 5-8 GB of free memory on each server, I would assume that the entire index has been paged into memory (I had worried about disk performance previously, as we are working in a virtualized environment). A stats query on the index in questions shows that the index is composed of > 7000 segments. This seemed high to me, but maybe it's appropriate. > Regardless, I dispatched an optimize command, but I am not seeing any > progress and the command has not returned. Current merges remains at zero, > and the segment count is not changing. Checking out hot threads in > ElasticHQ, I initially saw an optimize call in the stack that was blocked > on a waitForMerge call. This however has disappeared, and I'm seeing no > evidence that the optimize is occuring. Does any of this seem out of the norm or unusual? Has anyone else had similar issues. This is the second time I have tried to optimize an index since upgrading. I've gotten the same result both time. Thanks in advance for any help/tips! - Elliott -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aa7777fe-0a07-4991-bcb8-8c5f94118eb7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa7777fe-0a07-4991-bcb8-8c5f94118eb7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 3, 2014, 3:21pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/2 "2014-04-03T15:21:29Z")
OK. Optimize finally returned, so I suppose something was happening in the background, but I'm still seeing over 6500 segments. Even after setting max\_num\_segments=5. Does this seem right? Queries are a little faster (350-400ms) but still not great. Bigdesk is still showing a fair amount of file IO. On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > Hi All, > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node cluster, > each with 64G of ram, with 24G allocated to Elasticsearch on each. I've > batch loaded approximately 86 million documents into a single index (4 > shards) and have started benchmarking cross\_field/multi\_match queries on > them. The index has one replica and takes up a total of 111G. I've run > several batches of warming queries, but queries are not as fast as I had > hoped, approximately 400-500ms each. Given that \*top \*(on Centos) shows > 5-8 GB of free memory on each server, I would assume that the entire index > has been paged into memory (I had worried about disk performance > previously, as we are working in a virtualized environment). > > A stats query on the index in questions shows that the index is composed > of \> 7000 segments. This seemed high to me, but maybe it's appropriate. > Regardless, I dispatched an optimize command, but I am not seeing any > progress and the command has not returned. Current merges remains at zero, > and the segment count is not changing. Checking out hot threads in > ElasticHQ, I initially saw an optimize call in the stack that was blocked > on a waitForMerge call. This however has disappeared, and I'm seeing no > evidence that the optimize is occuring. > > Does any of this seem out of the norm or unusual? Has anyone else had > similar issues. This is the second time I have tried to optimize an index > since upgrading. I've gotten the same result both time. > > Thanks in advance for any help/tips! > > - Elliott -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/01ad85fc-3ad0-4c90-9f6e-1ed275bd7312%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/01ad85fc-3ad0-4c90-9f6e-1ed275bd7312%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 4, 2014, 3:27pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/3 "2014-04-04T15:27:29Z")
Any thoughts on this? I've run optimize several more times, and the number of segments falls each time, but I'm still over 1000 segments per shard. Has anyone else run into something similar? On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > OK. Optimize finally returned, so I suppose something was happening in > the background, but I'm still seeing over 6500 segments. Even after > setting max\_num\_segments=5. Does this seem right? Queries are a little > faster (350-400ms) but still not great. Bigdesk is still showing a fair > amount of file IO. > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > Hi All, > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > each. I've batch loaded approximately 86 million documents into a single > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > queries on them. The index has one replica and takes up a total of 111G. > > I've run several batches of warming queries, but queries are not as fast as > > I had hoped, approximately 400-500ms each. Given that \*top \*(on Centos) > > shows 5-8 GB of free memory on each server, I would assume that the entire > > index has been paged into memory (I had worried about disk performance > > previously, as we are working in a virtualized environment). > > > > A stats query on the index in questions shows that the index is composed > > of \> 7000 segments. This seemed high to me, but maybe it's appropriate. > > Regardless, I dispatched an optimize command, but I am not seeing any > > progress and the command has not returned. Current merges remains at zero, > > and the segment count is not changing. Checking out hot threads in > > ElasticHQ, I initially saw an optimize call in the stack that was blocked > > on a waitForMerge call. This however has disappeared, and I'm seeing no > > evidence that the optimize is occuring. > > > > Does any of this seem out of the norm or unusual? Has anyone else had > > similar issues. This is the second time I have tried to optimize an index > > since upgrading. I've gotten the same result both time. > > > > Thanks in advance for any help/tips! > > > > - Elliott -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 4, 2014, 3:53pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/4 "2014-04-04T15:53:00Z")
Elasticsearch throttles merges by default so that they don't slow search down too much. This is usually preferable for read/writes loads, but in your case it looks like you batch-indexed a lot of documents at once and merges couldn't keep up with the indexing rate so you ended up with a very high number of segments. The thing is that merge throttling also applies to optimize calls, which might explain why your calls to the optimize API last forever. Could you try to disable merge throttling[1] before running a call to optimize again to see if the situation improves? [1] > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Fri, Apr 4, 2014 at 5:27 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > Any thoughts on this? I've run optimize several more times, and the > number of segments falls each time, but I'm still over 1000 segments per > shard. Has anyone else run into something similar? > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > OK. Optimize finally returned, so I suppose something was happening in > > the background, but I'm still seeing over 6500 segments. Even after > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > amount of file IO. > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > Hi All, > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > each. I've batch loaded approximately 86 million documents into a single > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > queries on them. The index has one replica and takes up a total of 111G. > > > I've run several batches of warming queries, but queries are not as fast as > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > entire index has been paged into memory (I had worried about disk > > > performance previously, as we are working in a virtualized environment). > > > > > > A stats query on the index in questions shows that the index is composed > > > of \> 7000 segments. This seemed high to me, but maybe it's appropriate. > > > Regardless, I dispatched an optimize command, but I am not seeing any > > > progress and the command has not returned. Current merges remains at zero, > > > and the segment count is not changing. Checking out hot threads in > > > ElasticHQ, I initially saw an optimize call in the stack that was blocked > > > on a waitForMerge call. This however has disappeared, and I'm seeing no > > > evidence that the optimize is occuring. > > > > > > Does any of this seem out of the norm or unusual? Has anyone else had > > > similar issues. This is the second time I have tried to optimize an index > > > since upgrading. I've gotten the same result both time. > > > > > > Thanks in advance for any help/tips! > > > > > > - Elliott > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6WMRx8x-rJJi3KS2CZUu9wSbX8Vmuy48CpHFM\_jUCXdw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6WMRx8x-rJJi3KS2CZUu9wSbX8Vmuy48CpHFM_jUCXdw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 4, 2014, 4:22pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/5 "2014-04-04T16:22:42Z")
Adrien, What you're saying makes sense. I batch load the records very quickly, indexing about 100g of data in a little over an hour. However, I've tried your suggestions and am not seeing any improvement (I set index.store.throttle.max\_bytes\_per\_sec = 500mb). File system reads and writes are flatlined, CPU usage is 0. The optimize command is just waiting to return. When I first kick off the optimize I occasionally see a very brief burst in disk writes on the node on which the command is called, but that's it. Still stumped. On Fri, Apr 4, 2014 at 11:53 AM, Adrien Grand \< [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote: > Elasticsearch throttles merges by default so that they don't slow search > down too much. This is usually preferable for read/writes loads, but in > your case it looks like you batch-indexed a lot of documents at once and > merges couldn't keep up with the indexing rate so you ended up with a very > high number of segments. The thing is that merge throttling also applies to > optimize calls, which might explain why your calls to the optimize API last > forever. > > Could you try to disable merge throttling[1] before running a call to > optimize again to see if the situation improves? > > [1] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-throttling) > > On Fri, Apr 4, 2014 at 5:27 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > > > Any thoughts on this? I've run optimize several more times, and the > > number of segments falls each time, but I'm still over 1000 segments per > > shard. Has anyone else run into something similar? > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > > > OK. Optimize finally returned, so I suppose something was happening in > > > the background, but I'm still seeing over 6500 segments. Even after > > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > > amount of file IO. > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > Hi All, > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > each. I've batch loaded approximately 86 million documents into a single > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > I've run several batches of warming queries, but queries are not as fast as > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > entire index has been paged into memory (I had worried about disk > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > A stats query on the index in questions shows that the index is > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > seeing any progress and the command has not returned. Current merges > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > seeing no evidence that the optimize is occuring. > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone else had > > > > similar issues. This is the second time I have tried to optimize an index > > > > since upgrading. I've gotten the same result both time. > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > - Elliott > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6WMRx8x-rJJi3KS2CZUu9wSbX8Vmuy48CpHFM\_jUCXdw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6WMRx8x-rJJi3KS2CZUu9wSbX8Vmuy48CpHFM_jUCXdw%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6WMRx8x-rJJi3KS2CZUu9wSbX8Vmuy48CpHFM\_jUCXdw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6WMRx8x-rJJi3KS2CZUu9wSbX8Vmuy48CpHFM_jUCXdw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFv5-vhDC3JGvRVH%2BdtssgRD-QZK--k2%3D0xg\_eBAsbVb3w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFv5-vhDC3JGvRVH%2BdtssgRD-QZK--k2%3D0xg_eBAsbVb3w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [April 4, 2014, 4:26pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/6 "2014-04-04T16:26:03Z")
Have you tried max\_num\_segments=1 on your optimize? On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > Any thoughts on this? I've run optimize several more times, and the > number of segments falls each time, but I'm still over 1000 segments per > shard. Has anyone else run into something similar? > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > OK. Optimize finally returned, so I suppose something was happening in > > the background, but I'm still seeing over 6500 segments. Even after > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > amount of file IO. > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > Hi All, > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > each. I've batch loaded approximately 86 million documents into a single > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > queries on them. The index has one replica and takes up a total of 111G. > > > I've run several batches of warming queries, but queries are not as fast as > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > entire index has been paged into memory (I had worried about disk > > > performance previously, as we are working in a virtualized environment). > > > > > > A stats query on the index in questions shows that the index is composed > > > of \> 7000 segments. This seemed high to me, but maybe it's appropriate. > > > Regardless, I dispatched an optimize command, but I am not seeing any > > > progress and the command has not returned. Current merges remains at zero, > > > and the segment count is not changing. Checking out hot threads in > > > ElasticHQ, I initially saw an optimize call in the stack that was blocked > > > on a waitForMerge call. This however has disappeared, and I'm seeing no > > > evidence that the optimize is occuring. > > > > > > Does any of this seem out of the norm or unusual? Has anyone else had > > > similar issues. This is the second time I have tried to optimize an index > > > since upgrading. I've gotten the same result both time. > > > > > > Thanks in advance for any help/tips! > > > > > > - Elliott > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 4, 2014, 5:18pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/7 "2014-04-04T17:18:48Z")
Yes. I have run max\_num\_segments=1 every time. On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< [michael.sick@serenesoftware.com](mailto:michael.sick@serenesoftware.com)\> wrote: > Have you tried max\_num\_segments=1 on your optimize? > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > > > Any thoughts on this? I've run optimize several more times, and the > > number of segments falls each time, but I'm still over 1000 segments per > > shard. Has anyone else run into something similar? > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > > > OK. Optimize finally returned, so I suppose something was happening in > > > the background, but I'm still seeing over 6500 segments. Even after > > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > > amount of file IO. > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > Hi All, > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > each. I've batch loaded approximately 86 million documents into a single > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > I've run several batches of warming queries, but queries are not as fast as > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > entire index has been paged into memory (I had worried about disk > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > A stats query on the index in questions shows that the index is > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > seeing any progress and the command has not returned. Current merges > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > seeing no evidence that the optimize is occuring. > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone else had > > > > similar issues. This is the second time I have tried to optimize an index > > > > since upgrading. I've gotten the same result both time. > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > - Elliott > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 4, 2014, 7:24pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/8 "2014-04-04T19:24:47Z")
Did you see a message in the logs confirming that the setting has been updated? It would be interesting to see the output of hot threads[1] to see what your node is doing. [1] > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > Yes. I have run max\_num\_segments=1 every time. > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > [michael.sick@serenesoftware.com](mailto:michael.sick@serenesoftware.com)\> wrote: > > > Have you tried max\_num\_segments=1 on your optimize? > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > > > > > Any thoughts on this? I've run optimize several more times, and the > > > number of segments falls each time, but I'm still over 1000 segments per > > > shard. Has anyone else run into something similar? > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > OK. Optimize finally returned, so I suppose something was happening in > > > > the background, but I'm still seeing over 6500 segments. Even after > > > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > > > amount of file IO. > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > Hi All, > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > entire index has been paged into memory (I had worried about disk > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > seeing any progress and the command has not returned. Current merges > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone else had > > > > > similar issues. This is the second time I have tried to optimize an index > > > > > since upgrading. I've gotten the same result both time. > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > - Elliott > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 7, 2014, 12:56pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/9 "2014-04-07T12:56:38Z")
Adrian, I ran the following command: curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' and received a { "acknowledged" : "true" } response. The logs showed "cluster state updated". I did have to close my index prior to changing the setting and reopen afterward. I've since began another optimize, but again it doesn't look like much is happening. The optimize isn't returning and the total CPU usage on every node is holding at about 2% of a single core. I would copy a hot\_threads stack trace, but I'm unfortunately on a closed network and this isn't possible. I can tell you that refreshes of hot\_threads show vary little happening. The occasional [merge] thread (always in a LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing on a waitForMerge() call) thread shows up, but it's always consuming 0-1% CPU. It sure feels like something isn't right. Any thoughts? On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \<[adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com) > wrote: > Did you see a message in the logs confirming that the setting has been > updated? It would be interesting to see the output of hot threads[1] to see > what your node is doing. > > [1] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > > > Yes. I have run max\_num\_segments=1 every time. > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > [michael.sick@serenesoftware.com](mailto:michael.sick@serenesoftware.com)\> wrote: > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > > > > > > > Any thoughts on this? I've run optimize several more times, and the > > > > number of segments falls each time, but I'm still over 1000 segments per > > > > shard. Has anyone else run into something similar? > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > OK. Optimize finally returned, so I suppose something was happening > > > > > in the background, but I'm still seeing over 6500 segments. Even after > > > > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > > > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > > > > amount of file IO. > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone else > > > > > > had similar issues. This is the second time I have tried to optimize an > > > > > > index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > - Elliott > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFv00f0zXQ-pd77n9K5%3DqCE4rtp%3DXV2HLqREKsF%2Bf8hbXg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFv00f0zXQ-pd77n9K5%3DqCE4rtp%3DXV2HLqREKsF%2Bf8hbXg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 9, 2014, 12:38pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/10 "2014-04-09T12:38:41Z")
Any other thoughts on this? Would 1500 segments per shard be significantly impacting performance? Have you guys noticed this behavior elsewhere? Thanks. On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > Adrian, > > I ran the following command: > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > and received a { "acknowledged" : "true" } response. The logs showed > "cluster state updated". > > I did have to close my index prior to changing the setting and reopen > afterward. > > I've since began another optimize, but again it doesn't look like much is > happening. The optimize isn't returning and the total CPU usage on every > node is holding at about 2% of a single core. I would copy a hot\_threads > stack trace, but I'm unfortunately on a closed network and this isn't > possible. I can tell you that refreshes of hot\_threads show vary little > happening. The occasional [merge] thread (always in a > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing on a > waitForMerge() call) thread shows up, but it's always consuming 0-1% CPU. > It sure feels like something isn't right. Any thoughts? > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \< > [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote: > > > Did you see a message in the logs confirming that the setting has been > > updated? It would be interesting to see the output of hot threads[1] to see > > what your node is doing. > > > > [1] > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > [michael.sick@serenesoftware.com](mailto:michael.sick@serenesoftware.com)\> wrote: > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<[ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com) > > > > > > > > > wrote: > > > > > > > > > Any thoughts on this? I've run optimize several more times, and the > > > > > number of segments falls each time, but I'm still over 1000 segments per > > > > > shard. Has anyone else run into something similar? > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was happening > > > > > > in the background, but I'm still seeing over 6500 segments. Even after > > > > > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > > > > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > > > > > amount of file IO. > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone else > > > > > > > had similar issues. This is the second time I have tried to optimize an > > > > > > > index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Adrien Grand > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 9, 2014, 12:56pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/11 "2014-04-09T12:56:35Z")
Hi Elliott, 1500 segments per shard is certainly way too much, and it is not normal that optimize doesn't manage to reduce the number of segments. - Is there anything suspicious in the logs? - Have you customized the merge policy or scheduler?[1] - Does the issue still reproduce if you restart your cluster? [1] > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > Any other thoughts on this? Would 1500 segments per shard be > significantly impacting performance? Have you guys noticed this behavior > elsewhere? > > Thanks. > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > Adrian, > > > > I ran the following command: > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > and received a { "acknowledged" : "true" } response. The logs showed > > "cluster state updated". > > > > I did have to close my index prior to changing the setting and reopen > > afterward. > > > > I've since began another optimize, but again it doesn't look like much is > > happening. The optimize isn't returning and the total CPU usage on every > > node is holding at about 2% of a single core. I would copy a hot\_threads > > stack trace, but I'm unfortunately on a closed network and this isn't > > possible. I can tell you that refreshes of hot\_threads show vary little > > happening. The occasional [merge] thread (always in a LinkedTransferQueue.awaitMatch() > > state) or [optimize] (doing nothing on a waitForMerge() call) thread shows > > up, but it's always consuming 0-1% CPU. It sure feels like something isn't > > right. Any thoughts? > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \ > com\> wrote: > > > > > Did you see a message in the logs confirming that the setting has been > > > updated? It would be interesting to see the output of hot threads[1] to see > > > what your node is doing. > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > reference/current/cluster-nodes-hot-threads.html > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > [michael.sick@serenesoftware.com](mailto:michael.sick@serenesoftware.com)\> wrote: > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \< > > > > > [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)\> wrote: > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, and the > > > > > > number of segments falls each time, but I'm still over 1000 segments per > > > > > > shard. Has anyone else run into something similar? > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was happening > > > > > > > in the background, but I'm still seeing over 6500 segments. Even after > > > > > > > setting max\_num\_segments=5. Does this seem right? Queries are a little > > > > > > > faster (350-400ms) but still not great. Bigdesk is still showing a fair > > > > > > > amount of file IO. > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone else > > > > > > > > had similar issues. This is the second time I have tried to optimize an > > > > > > > > index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-](https://groups.google.com/d/msgid/elasticsearch/5391291f-) > > > > > > > 5c5e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/CAP8axnD7BUziGct2%3Db% > > > > > 3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW% > > > > 3Demc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > Adrien Grand > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8% > > > 2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4D577J%2B0NvmUwq7Kh049ZBW2hu8vGnzrZFefwazc1Eow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4D577J%2B0NvmUwq7Kh049ZBW2hu8vGnzrZFefwazc1Eow%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 9, 2014, 2:06pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/12 "2014-04-09T14:06:57Z")
Hi Adrien, I did customize my merge policy, although I did so only because I was so surprised by the number of segments left over after the load. I'm pretty sure the optimize problem was happening before I made this change, but either way here are my settings: "index" : { "merge" : { "policy" : { "max\_merged\_segment" : "20gb", "segments\_per\_tier" : 5, "floor\_segment" : "10mb" }, "scheduler" : "concurrentmergescheduler" } } Not sure whether this set up could be a contributing factor or not. Nothing really jumps out at me in the logs. In fact, when i kick off the optimize, I don't see any logging at all. Should I? I'm running the following command: curl -XPOST [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) Thanks! On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote: > Hi Elliott, > > 1500 segments per shard is certainly way too much, and it is not normal > that optimize doesn't manage to reduce the number of segments. > > - Is there anything suspicious in the logs? > - Have you customized the merge policy or scheduler?[1] > - Does the issue still reproduce if you restart your cluster? > > [1] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html) > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com)\ > > > wrote: > > > Any other thoughts on this? Would 1500 segments per shard be > > significantly impacting performance? Have you guys noticed this behavior > > elsewhere? > > > > Thanks. > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > > > Adrian, > > > > > > I ran the following command: > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > > > and received a { "acknowledged" : "true" } response. The logs showed > > > "cluster state updated". > > > > > > I did have to close my index prior to changing the setting and reopen > > > afterward. > > > > > > I've since began another optimize, but again it doesn't look like much > > > is happening. The optimize isn't returning and the total CPU usage on > > > every node is holding at about 2% of a single core. I would copy a > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this > > > isn't possible. I can tell you that refreshes of hot\_threads show vary > > > little happening. The occasional [merge] thread (always in a > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing on > > > a waitForMerge() call) thread shows up, but it's always consuming 0-1% > > > CPU. It sure feels like something isn't right. Any thoughts? > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \ > > com \\> wrote: > > > > > > > Did you see a message in the logs confirming that the setting has been > > > > updated? It would be interesting to see the output of hot threads[1] to see > > > > what your node is doing. > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > > reference/current/cluster-nodes-hot-threads.html > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com)\ > > > > > > > > > wrote: > > > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com) \\> wrote: > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com)\ > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, and the > > > > > > > number of segments falls each time, but I'm still over 1000 segments per > > > > > > > shard. Has anyone else run into something similar? > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a > > > > > > > > fair amount of file IO. > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > > > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone else > > > > > > > > > had similar issues. This is the second time I have tried to optimize an > > > > > > > > > index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-](https://groups.google.com/d/msgid/elasticsearch/5391291f-) > > > > > > > > 5c5e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/CAP8axnD7BUziGct2%3Db% > > > > > > 3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW% > > > > > 3Demc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > Adrien Grand > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8% > > > > 2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/775e2e32-5891-4514-bd53-32b52c0deb1a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/775e2e32-5891-4514-bd53-32b52c0deb1a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 9, 2014, 3:10pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/13 "2014-04-09T15:10:56Z")
Hi Adrien, I kept the logs up over the last optimize call, and I did see an exception. I Ctrl-C'd a curl optimize call before making another one, but I don't think that that caused this exception. The error is essentially as follows: netty - Caught exception while handling client http traffic, closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200] java.nio.channels.ClosedChannelException at AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at AbstractNioWorker.writeFromUserCode at NioServerSocketPipelineSink.handleAcceptedSocket at NioServerSocketPipelineSink.eventSunk at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream at Channels.write at OneToOneEncoder.doEncode at OneToOneEncoder.handleDownstream at DefaultChannelPipeline.sendDownstream at DefaultChannelPipeline.sendDownstream at Channels.write at AbstractChannel.write at NettyHttpChannel.sendResponse at RestOptimizeAction$1.onResponse(95) at RestOptimizeAction$1.onResponse(85) at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run Sorry about the crappy stack trace. Still, looks like this might point to a problem! The exception fired about an hour after I kicked off the optimize. Any thoughts? On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote: > Hi Adrien, > > I did customize my merge policy, although I did so only because I was so > surprised by the number of segments left over after the load. I'm pretty > sure the optimize problem was happening before I made this change, but > either way here are my settings: > > "index" : { > "merge" : { > "policy" : { > "max\_merged\_segment" : "20gb", > "segments\_per\_tier" : 5, > "floor\_segment" : "10mb" > }, > "scheduler" : "concurrentmergescheduler" > } > } > > Not sure whether this set up could be a contributing factor or not. > Nothing really jumps out at me in the logs. In fact, when i kick off the > optimize, I don't see any logging at all. Should I? > > I'm running the following command: curl -XPOST > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) > > Thanks! > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote: > > > Hi Elliott, > > > > 1500 segments per shard is certainly way too much, and it is not normal > > that optimize doesn't manage to reduce the number of segments. > > > > - Is there anything suspicious in the logs? > > - Have you customized the merge policy or scheduler?[1] > > - Does the issue still reproduce if you restart your cluster? > > > > [1] > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html) > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > Any other thoughts on this? Would 1500 segments per shard be > > > significantly impacting performance? Have you guys noticed this behavior > > > elsewhere? > > > > > > Thanks. > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > Adrian, > > > > > > > > I ran the following command: > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > > > > > and received a { "acknowledged" : "true" } response. The logs showed > > > > "cluster state updated". > > > > > > > > I did have to close my index prior to changing the setting and reopen > > > > afterward. > > > > > > > > I've since began another optimize, but again it doesn't look like much > > > > is happening. The optimize isn't returning and the total CPU usage on > > > > every node is holding at about 2% of a single core. I would copy a > > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this > > > > isn't possible. I can tell you that refreshes of hot\_threads show vary > > > > little happening. The occasional [merge] thread (always in a > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing > > > > on a waitForMerge() call) thread shows up, but it's always consuming 0-1% > > > > CPU. It sure feels like something isn't right. Any thoughts? > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \ > > > com\> wrote: > > > > > > > > > Did you see a message in the logs confirming that the setting has been > > > > > updated? It would be interesting to see the output of hot threads[1] to see > > > > > what your node is doing. > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > > > reference/current/cluster-nodes-hot-threads.html > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote: > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \< > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, and > > > > > > > > the number of segments falls each time, but I'm still over 1000 segments > > > > > > > > per shard. Has anyone else run into something similar? > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw > > > > > > > > wrote: > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a > > > > > > > > > fair amount of file IO. > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node > > > > > > > > > > cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone > > > > > > > > > > else had similar issues. This is the second time I have tried to optimize > > > > > > > > > > an index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-](https://groups.google.com/d/msgid/elasticsearch/5391291f-) > > > > > > > > > 5c5e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/) > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%](https://groups.google.com/d/msgid/elasticsearch/CAGCt%25) > > > > > > 2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail. > > > > > > [gmail.com](http://gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > Adrien Grand > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ% > > > > > 3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5189de86-30ab-4c73-acc8-6831058652d9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5189de86-30ab-4c73-acc8-6831058652d9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 9, 2014, 3:20pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/14 "2014-04-09T15:20:36Z")
The exception is just a side effect because you pressed ctrl-c and the response could not be transmitted back, it does not point to the problem. You should use [http://localhost:9200/index/\_optimize?max\_num\_segments=1](http://localhost:9200/index/_optimize?max_num_segments=1) instead of [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFJw%3D7MDw8EW0DtnpqzVturkb9UmdyiiZPUJ8m-3HaMg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFJw%3D7MDw8EW0DtnpqzVturkb9UmdyiiZPUJ8m-3HaMg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 9, 2014, 3:45pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/15 "2014-04-09T15:45:27Z")
Thanks Jorg. That makes sense. I am actually using max\_num\_segments=1, just forgot to add it... On Wed, Apr 9, 2014 at 11:20 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > The exception is just a side effect because you pressed ctrl-c and the > response could not be transmitted back, it does not point to the problem. > > You should use > > [http://localhost:9200/index/\_optimize?max\_num\_segments=1](http://localhost:9200/index/_optimize?max_num_segments=1) > > instead of > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) > > Jörg > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFJw%3D7MDw8EW0DtnpqzVturkb9UmdyiiZPUJ8m-3HaMg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFJw%3D7MDw8EW0DtnpqzVturkb9UmdyiiZPUJ8m-3HaMg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFJw%3D7MDw8EW0DtnpqzVturkb9UmdyiiZPUJ8m-3HaMg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFJw%3D7MDw8EW0DtnpqzVturkb9UmdyiiZPUJ8m-3HaMg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFuGLJaBLfB1PyFWKRS8HBJ81%2BwN1BFua6Ua5H7QyeVybQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFuGLJaBLfB1PyFWKRS8HBJ81%2BwN1BFua6Ua5H7QyeVybQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chuck](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@chuck](https://discuss.elastic.co/u/chuck)\ **Post date:** [April 10, 2014, 12:10pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/16 "2014-04-10T12:10:27Z")
Adrien, Just an FYI, after resetting the cluster, things seem to have improved. Optimize calls now lead to CPU/IO activity over their duration. Max\_num\_segments=1 does not seem to be working for me on any given call, as each call would only reduce the segment count by about 600-700. I ran 10 calls in sequence overnight, and actually got down to 4 segments (1/shard)! I'm glad I got the index optimized, searches are literally 10-20 times faster without 1500/segments per shard to deal with. It's awesome. That said, any thoughts on why the index wasn't merging on its own, or why optimize was returning prematurely? On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote: > Hi Adrien, > > I kept the logs up over the last optimize call, and I did see an > exception. I Ctrl-C'd a curl optimize call before making another one, but > I don't think that that caused this exception. The error is essentially as > follows: > > netty - Caught exception while handling client http traffic, closing > connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200] > > java.nio.channels.ClosedChannelException at > AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) > at AbstractNioWorker.writeFromUserCode > at NioServerSocketPipelineSink.handleAcceptedSocket > at NioServerSocketPipelineSink.eventSunk > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream > at Channels.write > at OneToOneEncoder.doEncode > at OneToOneEncoder.handleDownstream > at DefaultChannelPipeline.sendDownstream > at DefaultChannelPipeline.sendDownstream > at Channels.write > at AbstractChannel.write > at NettyHttpChannel.sendResponse > at RestOptimizeAction$1.onResponse(95) > at RestOptimizeAction$1.onResponse(85) > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run > > Sorry about the crappy stack trace. Still, looks like this might point to > a problem! The exception fired about an hour after I kicked off the > optimize. Any thoughts? > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote: > > > Hi Adrien, > > > > I did customize my merge policy, although I did so only because I was so > > surprised by the number of segments left over after the load. I'm pretty > > sure the optimize problem was happening before I made this change, but > > either way here are my settings: > > > > "index" : { > > "merge" : { > > "policy" : { > > "max\_merged\_segment" : "20gb", > > "segments\_per\_tier" : 5, > > "floor\_segment" : "10mb" > > }, > > "scheduler" : "concurrentmergescheduler" > > } > > } > > > > Not sure whether this set up could be a contributing factor or not. > > Nothing really jumps out at me in the logs. In fact, when i kick off the > > optimize, I don't see any logging at all. Should I? > > > > I'm running the following command: curl -XPOST > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) > > > > Thanks! > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote: > > > > > Hi Elliott, > > > > > > 1500 segments per shard is certainly way too much, and it is not normal > > > that optimize doesn't manage to reduce the number of segments. > > > > > > - Is there anything suspicious in the logs? > > > - Have you customized the merge policy or scheduler?[1] > > > - Does the issue still reproduce if you restart your cluster? > > > > > > [1] > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html) > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > > > Any other thoughts on this? Would 1500 segments per shard be > > > > significantly impacting performance? Have you guys noticed this behavior > > > > elsewhere? > > > > > > > > Thanks. > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > Adrian, > > > > > > > > > > I ran the following command: > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs showed > > > > > "cluster state updated". > > > > > > > > > > I did have to close my index prior to changing the setting and reopen > > > > > afterward. > > > > > > > > > > I've since began another optimize, but again it doesn't look like much > > > > > is happening. The optimize isn't returning and the total CPU usage on > > > > > every node is holding at about 2% of a single core. I would copy a > > > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this > > > > > isn't possible. I can tell you that refreshes of hot\_threads show vary > > > > > little happening. The occasional [merge] thread (always in a > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing > > > > > on a waitForMerge() call) thread shows up, but it's always consuming 0-1% > > > > > CPU. It sure feels like something isn't right. Any thoughts? > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \ > > > > com\> wrote: > > > > > > > > > > > Did you see a message in the logs confirming that the setting has > > > > > > been updated? It would be interesting to see the output of hot threads[1] > > > > > > to see what your node is doing. > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > > > > reference/current/cluster-nodes-hot-threads.html > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote: > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \< > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, and > > > > > > > > > the number of segments falls each time, but I'm still over 1000 segments > > > > > > > > > per shard. Has anyone else run into something similar? > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a > > > > > > > > > > fair amount of file IO. > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 > > > > > > > > > > > node cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > > > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone > > > > > > > > > > > else had similar issues. This is the second time I have tried to optimize > > > > > > > > > > > an index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-](https://groups.google.com/d/msgid/elasticsearch/5391291f-) > > > > > > > > > > 5c5e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/) > > > > > > > > CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO > > > > > > > > DFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%](https://groups.google.com/d/msgid/elasticsearch/CAGCt%25) > > > > > > > 2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail. > > > > > > > [gmail.com](http://gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > Adrien Grand > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ% > > > > > > 3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 10, 2014, 11:35pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/17 "2014-04-10T23:35:42Z")
Thanks for reporting this, the behavior is definitely unexpected. I'll test \_optimize on very large numbers of shards to see if I can reproduce the issue. On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw [ebradshaw1@gmail.com](mailto:ebradshaw1@gmail.com)wrote: > Adrien, > > Just an FYI, after resetting the cluster, things seem to have improved. > Optimize calls now lead to CPU/IO activity over their duration. > Max\_num\_segments=1 does not seem to be working for me on any given call, as > each call would only reduce the segment count by about 600-700. I ran 10 > calls in sequence overnight, and actually got down to 4 segments (1/shard)! > > I'm glad I got the index optimized, searches are literally 10-20 times > faster without 1500/segments per shard to deal with. It's awesome. > > That said, any thoughts on why the index wasn't merging on its own, or why > optimize was returning prematurely? > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote: > > > Hi Adrien, > > > > I kept the logs up over the last optimize call, and I did see an > > exception. I Ctrl-C'd a curl optimize call before making another one, but > > I don't think that that caused this exception. The error is essentially as > > follows: > > > > netty - Caught exception while handling client http traffic, closing > > connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200] > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker. > > cleanUpWriteBuffer(AbstractNioWorker.java:433) > > at AbstractNioWorker.writeFromUserCode > > at NioServerSocketPipelineSink.handleAcceptedSocket > > at NioServerSocketPipelineSink.eventSunk > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream > > at Channels.write > > at OneToOneEncoder.doEncode > > at OneToOneEncoder.handleDownstream > > at DefaultChannelPipeline.sendDownstream > > at DefaultChannelPipeline.sendDownstream > > at Channels.write > > at AbstractChannel.write > > at NettyHttpChannel.sendResponse > > at RestOptimizeAction$1.onResponse(95) > > at RestOptimizeAction$1.onResponse(85) > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run > > > > Sorry about the crappy stack trace. Still, looks like this might point > > to a problem! The exception fired about an hour after I kicked off the > > optimize. Any thoughts? > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote: > > > > > Hi Adrien, > > > > > > I did customize my merge policy, although I did so only because I was so > > > surprised by the number of segments left over after the load. I'm pretty > > > sure the optimize problem was happening before I made this change, but > > > either way here are my settings: > > > > > > "index" : { > > > "merge" : { > > > "policy" : { > > > "max\_merged\_segment" : "20gb", > > > "segments\_per\_tier" : 5, > > > "floor\_segment" : "10mb" > > > }, > > > "scheduler" : "concurrentmergescheduler" > > > } > > > } > > > > > > Not sure whether this set up could be a contributing factor or not. > > > Nothing really jumps out at me in the logs. In fact, when i kick off the > > > optimize, I don't see any logging at all. Should I? > > > > > > I'm running the following command: curl -XPOST > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) > > > > > > Thanks! > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote: > > > > > > > Hi Elliott, > > > > > > > > 1500 segments per shard is certainly way too much, and it is not normal > > > > that optimize doesn't manage to reduce the number of segments. > > > > > > > > - Is there anything suspicious in the logs? > > > > - Have you customized the merge policy or scheduler?[1] > > > > - Does the issue still reproduce if you restart your cluster? > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > > reference/current/index-modules-merge.html > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be > > > > > significantly impacting performance? Have you guys noticed this behavior > > > > > elsewhere? > > > > > > > > > > Thanks. > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > Adrian, > > > > > > > > > > > > I ran the following command: > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs showed > > > > > > "cluster state updated". > > > > > > > > > > > > I did have to close my index prior to changing the setting and reopen > > > > > > afterward. > > > > > > > > > > > > I've since began another optimize, but again it doesn't look like > > > > > > much is happening. The optimize isn't returning and the total CPU usage on > > > > > > every node is holding at about 2% of a single core. I would copy a > > > > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this > > > > > > isn't possible. I can tell you that refreshes of hot\_threads show vary > > > > > > little happening. The occasional [merge] thread (always in a > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing > > > > > > on a waitForMerge() call) thread shows up, but it's always consuming 0-1% > > > > > > CPU. It sure feels like something isn't right. Any thoughts? > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \< > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote: > > > > > > > > > > > > > Did you see a message in the logs confirming that the setting has > > > > > > > been updated? It would be interesting to see the output of hot threads[1] > > > > > > > to see what your node is doing. > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc) > > > > > > > e/current/cluster-nodes-hot-threads.html > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com) > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote: > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \< > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, and > > > > > > > > > > the number of segments falls each time, but I'm still over 1000 segments > > > > > > > > > > per shard. Has anyone else run into something similar? > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was > > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even > > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a > > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a > > > > > > > > > > > fair amount of file IO. > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 > > > > > > > > > > > > node cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index is > > > > > > > > > > > > composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone > > > > > > > > > > > > else had similar issues. This is the second time I have tried to optimize > > > > > > > > > > > > an index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5) > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz) > > > > > > > > > iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS) > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > Adrien Grand > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/to](https://groups.google.com/d/to) > > > > > > > pic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP) > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6% > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > Adrien Grand > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6xdRmyK5cfCPtJUijqQ8nfV56RNLsUVXbYRdpY-ZqStA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6xdRmyK5cfCPtJUijqQ8nfV56RNLsUVXbYRdpY-ZqStA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robin\_Wallin](https://avatars.discourse-cdn.com/v4/letter/r/3be4f8/32.png) [@Robin\_Wallin](https://discuss.elastic.co/u/Robin_Wallin)\ **Post date:** [April 11, 2014, 7:45am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/18 "2014-04-11T07:45:18Z")
Hello, We are experiencing a related problem with 1.1.0. Segments do not seem to merge as they should during indexing. The optimize API does practically nothing in terms of lowering the segments count either. The problem persists through a cluster restart. The vast amount of segments seem to be greatly impact the performance of the cluster, in a very negative way. We currently have 414 million documents across 3 nodes, each shard has in average 1200 segments(!). With 1.0.1 we had even more documents, ~650 million, without any segment problems. Looking in Marvel we were hovering at around 30-40 segments per shard back then. Best Regards, Robin On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote: > Thanks for reporting this, the behavior is definitely unexpected. I'll > test \_optimize on very large numbers of shards to see if I can reproduce > the issue. > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com)\ > > > wrote: > > > Adrien, > > > > Just an FYI, after resetting the cluster, things seem to have improved. > > Optimize calls now lead to CPU/IO activity over their duration. > > Max\_num\_segments=1 does not seem to be working for me on any given call, as > > each call would only reduce the segment count by about 600-700. I ran 10 > > calls in sequence overnight, and actually got down to 4 segments (1/shard)! > > > > I'm glad I got the index optimized, searches are literally 10-20 times > > faster without 1500/segments per shard to deal with. It's awesome. > > > > That said, any thoughts on why the index wasn't merging on its own, or > > why optimize was returning prematurely? > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote: > > > > > Hi Adrien, > > > > > > I kept the logs up over the last optimize call, and I did see an > > > exception. I Ctrl-C'd a curl optimize call before making another one, but > > > I don't think that that caused this exception. The error is essentially as > > > follows: > > > > > > netty - Caught exception while handling client http traffic, closing > > > connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200] > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker. > > > cleanUpWriteBuffer(AbstractNioWorker.java:433) > > > at AbstractNioWorker.writeFromUserCode > > > at NioServerSocketPipelineSink.handleAcceptedSocket > > > at NioServerSocketPipelineSink.eventSunk > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream > > > at Channels.write > > > at OneToOneEncoder.doEncode > > > at OneToOneEncoder.handleDownstream > > > at DefaultChannelPipeline.sendDownstream > > > at DefaultChannelPipeline.sendDownstream > > > at Channels.write > > > at AbstractChannel.write > > > at NettyHttpChannel.sendResponse > > > at RestOptimizeAction$1.onResponse(95) > > > at RestOptimizeAction$1.onResponse(85) > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run > > > > > > Sorry about the crappy stack trace. Still, looks like this might point > > > to a problem! The exception fired about an hour after I kicked off the > > > optimize. Any thoughts? > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > Hi Adrien, > > > > > > > > I did customize my merge policy, although I did so only because I was > > > > so surprised by the number of segments left over after the load. I'm > > > > pretty sure the optimize problem was happening before I made this change, > > > > but either way here are my settings: > > > > > > > > "index" : { > > > > "merge" : { > > > > "policy" : { > > > > "max\_merged\_segment" : "20gb", > > > > "segments\_per\_tier" : 5, > > > > "floor\_segment" : "10mb" > > > > }, > > > > "scheduler" : "concurrentmergescheduler" > > > > } > > > > } > > > > > > > > Not sure whether this set up could be a contributing factor or not. > > > > Nothing really jumps out at me in the logs. In fact, when i kick off the > > > > optimize, I don't see any logging at all. Should I? > > > > > > > > I'm running the following command: curl -XPOST > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) > > > > > > > > Thanks! > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote: > > > > > > > > > Hi Elliott, > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is not > > > > > normal that optimize doesn't manage to reduce the number of segments. > > > > > > > > > > - Is there anything suspicious in the logs? > > > > > - Have you customized the merge policy or scheduler?[1] > > > > > - Does the issue still reproduce if you restart your cluster? > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > > > reference/current/index-modules-merge.html > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be > > > > > > significantly impacting performance? Have you guys noticed this behavior > > > > > > elsewhere? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > Adrian, > > > > > > > > > > > > > > I ran the following command: > > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs > > > > > > > showed "cluster state updated". > > > > > > > > > > > > > > I did have to close my index prior to changing the setting and > > > > > > > reopen afterward. > > > > > > > > > > > > > > I've since began another optimize, but again it doesn't look like > > > > > > > much is happening. The optimize isn't returning and the total CPU usage on > > > > > > > every node is holding at about 2% of a single core. I would copy a > > > > > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this > > > > > > > isn't possible. I can tell you that refreshes of hot\_threads show vary > > > > > > > little happening. The occasional [merge] thread (always in a > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any > > > > > > > thoughts? > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \< > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote: > > > > > > > > > > > > > > > Did you see a message in the logs confirming that the setting has > > > > > > > > been updated? It would be interesting to see the output of hot threads[1] > > > > > > > > to see what your node is doing. > > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc) > > > > > > > > e/current/cluster-nodes-hot-threads.html > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \< > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote: > > > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \< > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, and > > > > > > > > > > > the number of segments falls each time, but I'm still over 1000 segments > > > > > > > > > > > per shard. Has anyone else run into something similar? > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was > > > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even > > > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a > > > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a > > > > > > > > > > > > fair amount of file IO. > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 > > > > > > > > > > > > > node cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index > > > > > > > > > > > > > is composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone > > > > > > > > > > > > > else had similar issues. This is the second time I have tried to optimize > > > > > > > > > > > > > an index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5) > > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/) > > > > > > > > > > unsubscribe. > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz) > > > > > > > > > > iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS) > > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > Adrien Grand > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP) > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6% > > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > Adrien Grand > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 11, 2014, 9:46am UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/19 "2014-04-11T09:46:54Z")
I managed to reproduce the issue locally, I'm looking into it. On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com) wrote: > Hello, > > We are experiencing a related problem with 1.1.0. Segments do not seem to > merge as they should during indexing. The optimize API does practically > nothing in terms of lowering the segments count either. The problem > persists through a cluster restart. The vast amount of segments seem to be > greatly impact the performance of the cluster, in a very negative way. > > We currently have 414 million documents across 3 nodes, each shard has in > average 1200 segments(!). > > With 1.0.1 we had even more documents, ~650 million, without any segment > problems. Looking in Marvel we were hovering at around 30-40 segments per > shard back then. > > Best Regards, > Robin > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote: > > > Thanks for reporting this, the behavior is definitely unexpected. I'll > > test \_optimize on very large numbers of shards to see if I can reproduce > > the issue. > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > Adrien, > > > > > > Just an FYI, after resetting the cluster, things seem to have improved. > > > Optimize calls now lead to CPU/IO activity over their duration. > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as > > > each call would only reduce the segment count by about 600-700. I ran 10 > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)! > > > > > > I'm glad I got the index optimized, searches are literally 10-20 times > > > faster without 1500/segments per shard to deal with. It's awesome. > > > > > > That said, any thoughts on why the index wasn't merging on its own, or > > > why optimize was returning prematurely? > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > Hi Adrien, > > > > > > > > I kept the logs up over the last optimize call, and I did see an > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but > > > > I don't think that that caused this exception. The error is essentially as > > > > follows: > > > > > > > > netty - Caught exception while handling client http traffic, closing > > > > connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200] > > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker. > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433) > > > > at AbstractNioWorker.writeFromUserCode > > > > at NioServerSocketPipelineSink.handleAcceptedSocket > > > > at NioServerSocketPipelineSink.eventSunk > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream > > > > at Channels.write > > > > at OneToOneEncoder.doEncode > > > > at OneToOneEncoder.handleDownstream > > > > at DefaultChannelPipeline.sendDownstream > > > > at DefaultChannelPipeline.sendDownstream > > > > at Channels.write > > > > at AbstractChannel.write > > > > at NettyHttpChannel.sendResponse > > > > at RestOptimizeAction$1.onResponse(95) > > > > at RestOptimizeAction$1.onResponse(85) > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run > > > > > > > > Sorry about the crappy stack trace. Still, looks like this might point > > > > to a problem! The exception fired about an hour after I kicked off the > > > > optimize. Any thoughts? > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > Hi Adrien, > > > > > > > > > > I did customize my merge policy, although I did so only because I was > > > > > so surprised by the number of segments left over after the load. I'm > > > > > pretty sure the optimize problem was happening before I made this change, > > > > > but either way here are my settings: > > > > > > > > > > "index" : { > > > > > "merge" : { > > > > > "policy" : { > > > > > "max\_merged\_segment" : "20gb", > > > > > "segments\_per\_tier" : 5, > > > > > "floor\_segment" : "10mb" > > > > > }, > > > > > "scheduler" : "concurrentmergescheduler" > > > > > } > > > > > } > > > > > > > > > > Not sure whether this set up could be a contributing factor or not. > > > > > Nothing really jumps out at me in the logs. In fact, when i kick off the > > > > > optimize, I don't see any logging at all. Should I? > > > > > > > > > > I'm running the following command: curl -XPOST > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) > > > > > > > > > > Thanks! > > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote: > > > > > > > > > > > Hi Elliott, > > > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is not > > > > > > normal that optimize doesn't manage to reduce the number of segments. > > > > > > > > > > > > - Is there anything suspicious in the logs? > > > > > > - Have you customized the merge policy or scheduler?[1] > > > > > > - Does the issue still reproduce if you restart your cluster? > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc) > > > > > > e/current/index-modules-merge.html > > > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be > > > > > > > significantly impacting performance? Have you guys noticed this behavior > > > > > > > elsewhere? > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > > > Adrian, > > > > > > > > > > > > > > > > I ran the following command: > > > > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs > > > > > > > > showed "cluster state updated". > > > > > > > > > > > > > > > > I did have to close my index prior to changing the setting and > > > > > > > > reopen afterward. > > > > > > > > > > > > > > > > I've since began another optimize, but again it doesn't look like > > > > > > > > much is happening. The optimize isn't returning and the total CPU usage on > > > > > > > > every node is holding at about 2% of a single core. I would copy a > > > > > > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this > > > > > > > > isn't possible. I can tell you that refreshes of hot\_threads show vary > > > > > > > > little happening. The occasional [merge] thread (always in a > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing > > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any > > > > > > > > thoughts? > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \< > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote: > > > > > > > > > > > > > > > > > Did you see a message in the logs confirming that the setting has > > > > > > > > > been updated? It would be interesting to see the output of hot threads[1] > > > > > > > > > to see what your node is doing. > > > > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc) > > > > > > > > > e/current/cluster-nodes-hot-threads.html > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \< > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \< > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, > > > > > > > > > > > > and the number of segments falls each time, but I'm still over 1000 > > > > > > > > > > > > segments per shard. Has anyone else run into something similar? > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was > > > > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even > > > > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a > > > > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a > > > > > > > > > > > > > fair amount of file IO. > > > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 > > > > > > > > > > > > > > node cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top \*(on > > > > > > > > > > > > > > Centos) shows 5-8 GB of free memory on each server, I would assume that the > > > > > > > > > > > > > > entire index has been paged into memory (I had worried about disk > > > > > > > > > > > > > > performance previously, as we are working in a virtualized environment). > > > > > > > > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index > > > > > > > > > > > > > > is composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has anyone > > > > > > > > > > > > > > else had similar issues. This is the second time I have tried to optimize > > > > > > > > > > > > > > an index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5) > > > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/) > > > > > > > > > > > unsubscribe. > > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > > to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz) > > > > > > > > > > > iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS) > > > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gma > > > > > > > > > > [il.com](http://il.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Adrien Grand > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > > > topic/elasticsearch/kqTRRADQBwc/unsubscribe. > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP) > > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922) > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > Adrien Grand > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce% > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 11, 2014, 12:22pm UTC](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790/20 "2014-04-11T12:22:06Z")
For reference I'm also on 1.1.0 but I'm not seeing more segments then I expect. I see an average of ~28 per shard on an index I write to constantly. I don't write all that quickly, \< 50 updates a second. Nik On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand \< [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote: > I managed to reproduce the issue locally, I'm looking into it. > > On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin [walro467@gmail.com](mailto:walro467@gmail.com) wrote: > > > Hello, > > > > We are experiencing a related problem with 1.1.0. Segments do not seem to > > merge as they should during indexing. The optimize API does practically > > nothing in terms of lowering the segments count either. The problem > > persists through a cluster restart. The vast amount of segments seem to be > > greatly impact the performance of the cluster, in a very negative way. > > > > We currently have 414 million documents across 3 nodes, each shard has in > > average 1200 segments(!). > > > > With 1.0.1 we had even more documents, ~650 million, without any segment > > problems. Looking in Marvel we were hovering at around 30-40 segments per > > shard back then. > > > > Best Regards, > > Robin > > > > On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote: > > > > > Thanks for reporting this, the behavior is definitely unexpected. I'll > > > test \_optimize on very large numbers of shards to see if I can reproduce > > > the issue. > > > > > > On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw [ebrad...@gmail.com](mailto:ebrad...@gmail.com)wrote: > > > > > > > Adrien, > > > > > > > > Just an FYI, after resetting the cluster, things seem to have > > > > improved. Optimize calls now lead to CPU/IO activity over their duration. > > > > Max\_num\_segments=1 does not seem to be working for me on any given call, as > > > > each call would only reduce the segment count by about 600-700. I ran 10 > > > > calls in sequence overnight, and actually got down to 4 segments (1/shard)! > > > > > > > > I'm glad I got the index optimized, searches are literally 10-20 times > > > > faster without 1500/segments per shard to deal with. It's awesome. > > > > > > > > That said, any thoughts on why the index wasn't merging on its own, or > > > > why optimize was returning prematurely? > > > > > > > > On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > Hi Adrien, > > > > > > > > > > I kept the logs up over the last optimize call, and I did see an > > > > > exception. I Ctrl-C'd a curl optimize call before making another one, but > > > > > I don't think that that caused this exception. The error is essentially as > > > > > follows: > > > > > > > > > > netty - Caught exception while handling client http traffic, closing > > > > > connection [id: 0x4d8f1a90, /127.0.0.1:33480 :\> /127.0.0.1:9200] > > > > > > > > > > java.nio.channels.ClosedChannelException at AbstractNioWorker. > > > > > cleanUpWriteBuffer(AbstractNioWorker.java:433) > > > > > at AbstractNioWorker.writeFromUserCode > > > > > at NioServerSocketPipelineSink.handleAcceptedSocket > > > > > at NioServerSocketPipelineSink.eventSunk > > > > > at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream > > > > > at Channels.write > > > > > at OneToOneEncoder.doEncode > > > > > at OneToOneEncoder.handleDownstream > > > > > at DefaultChannelPipeline.sendDownstream > > > > > at DefaultChannelPipeline.sendDownstream > > > > > at Channels.write > > > > > at AbstractChannel.write > > > > > at NettyHttpChannel.sendResponse > > > > > at RestOptimizeAction$1.onResponse(95) > > > > > at RestOptimizeAction$1.onResponse(85) > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation > > > > > at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run > > > > > > > > > > Sorry about the crappy stack trace. Still, looks like this might > > > > > point to a problem! The exception fired about an hour after I kicked off > > > > > the optimize. Any thoughts? > > > > > > > > > > On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > Hi Adrien, > > > > > > > > > > > > I did customize my merge policy, although I did so only because I was > > > > > > so surprised by the number of segments left over after the load. I'm > > > > > > pretty sure the optimize problem was happening before I made this change, > > > > > > but either way here are my settings: > > > > > > > > > > > > "index" : { > > > > > > "merge" : { > > > > > > "policy" : { > > > > > > "max\_merged\_segment" : "20gb", > > > > > > "segments\_per\_tier" : 5, > > > > > > "floor\_segment" : "10mb" > > > > > > }, > > > > > > "scheduler" : "concurrentmergescheduler" > > > > > > } > > > > > > } > > > > > > > > > > > > Not sure whether this set up could be a contributing factor or not. > > > > > > Nothing really jumps out at me in the logs. In fact, when i kick off the > > > > > > optimize, I don't see any logging at all. Should I? > > > > > > > > > > > > I'm running the following command: curl -XPOST > > > > > > [http://localhost:9200/index/\_optimize](http://localhost:9200/index/_optimize) > > > > > > > > > > > > Thanks! > > > > > > > > > > > > On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote: > > > > > > > > > > > > > Hi Elliott, > > > > > > > > > > > > > > 1500 segments per shard is certainly way too much, and it is not > > > > > > > normal that optimize doesn't manage to reduce the number of segments. > > > > > > > > > > > > > > - Is there anything suspicious in the logs? > > > > > > > - Have you customized the merge policy or scheduler?[1] > > > > > > > - Does the issue still reproduce if you restart your cluster? > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc) > > > > > > > e/current/index-modules-merge.html > > > > > > > > > > > > > > On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw \<[ebrad...@gmail.com](mailto:ebrad...@gmail.com) > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Any other thoughts on this? Would 1500 segments per shard be > > > > > > > > significantly impacting performance? Have you guys noticed this behavior > > > > > > > > elsewhere? > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > > > > > > > > > > > > > > > > Adrian, > > > > > > > > > > > > > > > > > > I ran the following command: > > > > > > > > > > > > > > > > > > curl -XPUT [http://localhost:9200/\_settings](http://localhost:9200/_settings) -d > > > > > > > > > '{"indices.store.throttle.max\_bytes\_per\_sec" : "10gb"}' > > > > > > > > > > > > > > > > > > and received a { "acknowledged" : "true" } response. The logs > > > > > > > > > showed "cluster state updated". > > > > > > > > > > > > > > > > > > I did have to close my index prior to changing the setting and > > > > > > > > > reopen afterward. > > > > > > > > > > > > > > > > > > I've since began another optimize, but again it doesn't look like > > > > > > > > > much is happening. The optimize isn't returning and the total CPU usage on > > > > > > > > > every node is holding at about 2% of a single core. I would copy a > > > > > > > > > hot\_threads stack trace, but I'm unfortunately on a closed network and this > > > > > > > > > isn't possible. I can tell you that refreshes of hot\_threads show vary > > > > > > > > > little happening. The occasional [merge] thread (always in a > > > > > > > > > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing > > > > > > > > > nothing on a waitForMerge() call) thread shows up, but it's always > > > > > > > > > consuming 0-1% CPU. It sure feels like something isn't right. Any > > > > > > > > > thoughts? > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand \< > > > > > > > > > [adrien...@elasticsearch.com](mailto:adrien...@elasticsearch.com)\> wrote: > > > > > > > > > > > > > > > > > > > Did you see a message in the logs confirming that the setting has > > > > > > > > > > been updated? It would be interesting to see the output of hot threads[1] > > > > > > > > > > to see what your node is doing. > > > > > > > > > > > > > > > > > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/referenc) > > > > > > > > > > e/current/cluster-nodes-hot-threads.html > > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw \< > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > Yes. I have run max\_num\_segments=1 every time. > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick \< > > > > > > > > > > > [michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > > > Have you tried max\_num\_segments=1 on your optimize? > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw \< > > > > > > > > > > > > [ebrad...@gmail.com](mailto:ebrad...@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Any thoughts on this? I've run optimize several more times, > > > > > > > > > > > > > and the number of segments falls each time, but I'm still over 1000 > > > > > > > > > > > > > segments per shard. Has anyone else run into something similar? > > > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > OK. Optimize finally returned, so I suppose something was > > > > > > > > > > > > > > happening in the background, but I'm still seeing over 6500 segments. Even > > > > > > > > > > > > > > after setting max\_num\_segments=5. Does this seem right? Queries are a > > > > > > > > > > > > > > little faster (350-400ms) but still not great. Bigdesk is still showing a > > > > > > > > > > > > > > fair amount of file IO. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 > > > > > > > > > > > > > > > node cluster, each with 64G of ram, with 24G allocated to Elasticsearch on > > > > > > > > > > > > > > > each. I've batch loaded approximately 86 million documents into a single > > > > > > > > > > > > > > > index (4 shards) and have started benchmarking cross\_field/multi\_match > > > > > > > > > > > > > > > queries on them. The index has one replica and takes up a total of 111G. > > > > > > > > > > > > > > > I've run several batches of warming queries, but queries are not as fast as > > > > > > > > > > > > > > > I had hoped, approximately 400-500ms each. Given that \*top > > > > > > > > > > > > > > > \*(on Centos) shows 5-8 GB of free memory on each server, I > > > > > > > > > > > > > > > would assume that the entire index has been paged into memory (I had > > > > > > > > > > > > > > > worried about disk performance previously, as we are working in a > > > > > > > > > > > > > > > virtualized environment). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > A stats query on the index in questions shows that the index > > > > > > > > > > > > > > > is composed of \> 7000 segments. This seemed high to me, but maybe it's > > > > > > > > > > > > > > > appropriate. Regardless, I dispatched an optimize command, but I am not > > > > > > > > > > > > > > > seeing any progress and the command has not returned. Current merges > > > > > > > > > > > > > > > remains at zero, and the segment count is not changing. Checking out hot > > > > > > > > > > > > > > > threads in ElasticHQ, I initially saw an optimize call in the stack that > > > > > > > > > > > > > > > was blocked on a waitForMerge call. This however has disappeared, and I'm > > > > > > > > > > > > > > > seeing no evidence that the optimize is occuring. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Does any of this seem out of the norm or unusual? Has > > > > > > > > > > > > > > > anyone else had similar issues. This is the second time I have tried to > > > > > > > > > > > > > > > optimize an index since upgrading. I've gotten the same result both time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help/tips! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Elliott > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5) > > > > > > > > > > > > > > e-4088-a1f2-93272beef0bb%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/) > > > > > > > > > > > > unsubscribe. > > > > > > > > > > > > To unsubscribe from this group and all its topics, send an > > > > > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz) > > > > > > > > > > > > iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS) > > > > > > > > > > > QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gma > > > > > > > > > > > [il.com](http://il.com)[https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk\_A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Adrien Grand > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/](https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/) > > > > > > > > > > unsubscribe. > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP) > > > > > > > > > > jijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%[40mail.gmail.com](http://40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO\_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8742280e-922](https://groups.google.com/d/msgid/elasticsearch/8742280e-922) > > > > > > > > > f-4e91-bcb2-6096ca0165e6%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > Adrien Grand > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce% > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > Adrien Grand > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). [Next page](https://discuss.elastic.co/t/elasticsearch-1-1-0-optimize-broken/16790.md?page=2)