# Changing Merge Policy And Optimization **URL:** \ **Category:** Elasticsearch\ **Created:** [April 24, 2013, 7:58am UTC](https://discuss.elastic.co/t/changing-merge-policy-and-optimization/11679 "2013-04-24T07:58:46Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![tarang\_dawer](https://avatars.discourse-cdn.com/v4/letter/t/c0e974/32.png) [@tarang\_dawer](https://discuss.elastic.co/u/tarang_dawer)\ **Post date:** [April 24, 2013, 7:58am UTC](https://discuss.elastic.co/t/changing-merge-policy-and-optimization/11679/1 "2013-04-24T07:58:46Z")
Hi All I indexed some data in elasticsearch and have a index of around 18 Gb , with 10 Shards for the index, with 0 replicas and using the default "tiered" merge policy. I issued a Optimize command , the number of segments came down to 22-30 for each shard , which does not seems to go down any further , irrespective of whether i specify the max\_num\_segments in the optimize call or not. I am thinking of changing the merge policy to log\_doc, (after closing the index) , but not able to find some proper documentation . My main aim is improving the search performance , by bringing down the number of segments for each shard , which i am thinking to bring in effect by changing the merge policy and then issuing an optimize call. Also , the argument that is sent along the optimize call , max\_num\_segments , is it for max segments for a shard or an index ? i tried changing the settings with "index.merge\_policy":"log\_doc" and index.merge.policy":"log\_doc" but it accepts both the parameters , does not returns which one is valid and in effect , even if i specify the wrong input , i.e "loc\_doc\_123". Could somebody please help me resolve the issue ? Thanks Tarang Dawer -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Randall\_McRee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/randall_mcree/32/47177_2.png) [@Randall\_McRee](https://discuss.elastic.co/u/Randall_McRee)\ **Post date:** [April 25, 2013, 11:56pm UTC](https://discuss.elastic.co/t/changing-merge-policy-and-optimization/11679/2 "2013-04-25T23:56:47Z")
Lots of questions. First, performance is a big topic, why have you decided that merge optimization is the right way to achieve your goals? You may be dissapointed. Segment optimization does achieve a measurable benefit in our production setup but its behind a lot of other things that we tried, number of cpus, memory size, vm memory, query structure, etc. That said, the segment optimize works on shards--each shard is a lucene index and the optimize is passed on to lucene. You can often read the lucene documentation to understand better the tradeoffs that are happening underneath the elasticsearch covers. You usually need to issue the optimize _twice_. This is a bug as far as I can tell. It _never_ reports failure. I have not tried changing the merge policy on an existing index but I would not be surprised if you need to rebuild the index to get any benefit. Note that as soon as you add docs to your newly optimized index the benefits of optimization start to be lost...so this really only makes sense on a query-mostly index. Good luck, Randy On Wed, Apr 24, 2013 at 12:58 AM, Tarang Dawer [tarang.dawer@gmail.com](mailto:tarang.dawer@gmail.com)wrote: > Hi All > I indexed some data in elasticsearch and have a index of around 18 Gb , > with 10 Shards for the index, with 0 replicas and using the default > "tiered" merge policy. > I issued a Optimize command , the number of segments came down to 22-30 > for each shard , which does not seems to go down any further , irrespective > of whether i specify the max\_num\_segments in the optimize call or not. > > I am thinking of changing the merge policy to log\_doc, (after closing the > index) , but not able to find some proper documentation . > My main aim is improving the search performance , by bringing down the > number of segments for each shard , which i am thinking to bring in effect > by changing the merge policy and then issuing an optimize call. > > Also , the argument that is sent along the optimize call , > max\_num\_segments , is it for max segments for a shard or an index ? > > i tried changing the settings with "index.merge\_policy":"log\_doc" and > index.merge.policy":"log\_doc" > > but it accepts both the parameters , does not returns which one is valid > and in effect , even if i specify the wrong input , i.e "loc\_doc\_123". > > Could somebody please help me resolve the issue ? > > Thanks > Tarang Dawer > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [May 1, 2013, 12:05pm UTC](https://discuss.elastic.co/t/changing-merge-policy-and-optimization/11679/3 "2013-05-01T12:05:23Z")
@Randy is there a specific way to invoke optimize in your case, or is it a blank optimize call? On Friday, April 26, 2013 3:56:47 AM UTC+4, RKM wrote: > Lots of questions. First, performance is a big topic, why have you decided > that merge optimization is the right way to achieve your goals? You may be > dissapointed. > > Segment optimization does achieve a measurable benefit in our production > setup but its behind a lot of other things that we tried, number of cpus, > memory size, vm memory, query structure, etc. > > That said, the segment optimize works on shards--each shard is a lucene > index and the optimize is passed on to lucene. You can often read the > lucene documentation to understand better the tradeoffs that are happening > underneath the elasticsearch covers. > > You usually need to issue the optimize _twice_. This is a bug as far as I > can tell. It _never_ reports failure. > > I have not tried changing the merge policy on an existing index but I > would not be surprised if you need to rebuild the index to get any benefit. > > Note that as soon as you add docs to your newly optimized index the > benefits of optimization start to be lost...so this really only makes sense > on a query-mostly index. > > Good luck, > Randy > > On Wed, Apr 24, 2013 at 12:58 AM, Tarang Dawer \<[tarang...@gmail.com](mailto:tarang...@gmail.com)\ > > > wrote: > > > Hi All > > I indexed some data in elasticsearch and have a index of around 18 Gb , > > with 10 Shards for the index, with 0 replicas and using the default > > "tiered" merge policy. > > I issued a Optimize command , the number of segments came down to 22-30 > > for each shard , which does not seems to go down any further , irrespective > > of whether i specify the max\_num\_segments in the optimize call or not. > > > > I am thinking of changing the merge policy to log\_doc, (after closing the > > index) , but not able to find some proper documentation . > > My main aim is improving the search performance , by bringing down the > > number of segments for each shard , which i am thinking to bring in effect > > by changing the merge policy and then issuing an optimize call. > > > > Also , the argument that is sent along the optimize call , > > max\_num\_segments , is it for max segments for a shard or an index ? > > > > i tried changing the settings with "index.merge\_policy":"log\_doc" and > > index.merge.policy":"log\_doc" > > > > but it accepts both the parameters , does not returns which one is valid > > and in effect , even if i specify the wrong input , i.e "loc\_doc\_123". > > > > Could somebody please help me resolve the issue ? > > > > Thanks > > Tarang Dawer > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Randall\_McRee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/randall_mcree/32/47177_2.png) [@Randall\_McRee](https://discuss.elastic.co/u/Randall_McRee)\ **Post date:** [May 1, 2013, 10:11pm UTC](https://discuss.elastic.co/t/changing-merge-policy-and-optimization/11679/4 "2013-05-01T22:11:43Z")
I'm referring to optimize when setting max\_num\_segments, e.g. curl -XPOST "[http://server:9200/myindex/\_optimize?max\_num\_segments=3](http://server:9200/myindex/_optimize?max_num_segments=3)" On Wed, May 1, 2013 at 5:05 AM, Mo [mohammady.mahdy@gmail.com](mailto:mohammady.mahdy@gmail.com) wrote: > @Randy is there a specific way to invoke optimize in your case, or is it a > blank optimize call? > > On Friday, April 26, 2013 3:56:47 AM UTC+4, RKM wrote: > > > Lots of questions. First, performance is a big topic, why have you > > decided that merge optimization is the right way to achieve your goals? You > > may be dissapointed. > > > > Segment optimization does achieve a measurable benefit in our production > > setup but its behind a lot of other things that we tried, number of cpus, > > memory size, vm memory, query structure, etc. > > > > That said, the segment optimize works on shards--each shard is a lucene > > index and the optimize is passed on to lucene. You can often read the > > lucene documentation to understand better the tradeoffs that are happening > > underneath the elasticsearch covers. > > > > You usually need to issue the optimize _twice_. This is a bug as far as I > > can tell. It _never_ reports failure. > > > > I have not tried changing the merge policy on an existing index but I > > would not be surprised if you need to rebuild the index to get any benefit. > > > > Note that as soon as you add docs to your newly optimized index the > > benefits of optimization start to be lost...so this really only makes sense > > on a query-mostly index. > > > > Good luck, > > Randy > > > > On Wed, Apr 24, 2013 at 12:58 AM, Tarang Dawer [tarang...@gmail.com](mailto:tarang...@gmail.com)wrote: > > > > > Hi All > > > I indexed some data in elasticsearch and have a index of around 18 Gb , > > > with 10 Shards for the index, with 0 replicas and using the default > > > "tiered" merge policy. > > > I issued a Optimize command , the number of segments came down to 22-30 > > > for each shard , which does not seems to go down any further , irrespective > > > of whether i specify the max\_num\_segments in the optimize call or not. > > > > > > I am thinking of changing the merge policy to log\_doc, (after closing > > > the index) , but not able to find some proper documentation . > > > My main aim is improving the search performance , by bringing down the > > > number of segments for each shard , which i am thinking to bring in effect > > > by changing the merge policy and then issuing an optimize call. > > > > > > Also , the argument that is sent along the optimize call , > > > max\_num\_segments , is it for max segments for a shard or an index ? > > > > > > i tried changing the settings with "index.merge\_policy":"log\_doc" and > > > index.merge.policy":"log\_doc" > > > > > > but it accepts both the parameters , does not returns which one is valid > > > and in effect , even if i specify the wrong input , i.e "loc\_doc\_123". > > > > > > Could somebody please help me resolve the issue ? > > > > > > Thanks > > > Tarang Dawer > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:38am UTC](https://discuss.elastic.co/t/changing-merge-policy-and-optimization/11679/5 "2017-07-06T02:38:45Z")