# Merge/segment understanding **URL:** \ **Category:** Elasticsearch\ **Created:** [March 28, 2014, 2:07pm UTC](https://discuss.elastic.co/t/merge-segment-understanding/16679 "2014-03-28T14:07:51Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Han\_JU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/han_ju/32/962_2.png) [@Han\_JU](https://discuss.elastic.co/u/Han_JU)\ **Post date:** [March 28, 2014, 2:07pm UTC](https://discuss.elastic.co/t/merge-segment-understanding/16679/1 "2014-03-28T14:07:51Z")
Hi, We want to understand how segments are created during bulk indexing. Say we've set the following param: "index.translog.flush\_threshold\_ops" : "50000", "index.translog.flush\_threshold\_size": "300mb", So it means that ES will not flush until it gets 50000 operations (index in this case). As a result, there's always 50000 documents get flushed/committed to Lucene at a single time. So it's intuitive for us that Lucene will not create segments that has under 50000 documents. But in our benchmark with this settings, we found out that there's lots of segments with, say, ~3000 documents, and the segment's size is far less than 300mb (the flush threshold). My questions are: - How do these small segments get generated given that we flush 50000 documents at a time? - Does avoid generating small segments helps indexing speed and merge speed? We are using ElasticSearch v1.0.1 and we also set these when benchmarking: { "index":{ "merge.policy.max\_merge\_at\_once":"999", "merge.policy.segments\_per\_tier":"999", "refresh\_interval":"-1", } } Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/21ec20de-0a6d-4196-abf3-d12287544b7f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/21ec20de-0a6d-4196-abf3-d12287544b7f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 28, 2014, 7:32pm UTC](https://discuss.elastic.co/t/merge-segment-understanding/16679/2 "2014-03-28T19:32:46Z")
The indexing buffer could also fill up which will flush to a segment. Also the translog flush is not "exactly" deterministic, for example "index.translog.interval" determines how often to check if the translog needs to be flushed or not. Anyway, I wouldn't worry about it if I were you. About the merge, I'd probably leave the defaults alone unless you are absolutely sure changing them helps you. The more segments there are, the more time it could take to do a merge. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ead045d0-714a-4906-9d1a-c8f1bca59512%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ead045d0-714a-4906-9d1a-c8f1bca59512%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Han\_JU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/han_ju/32/962_2.png) [@Han\_JU](https://discuss.elastic.co/u/Han_JU)\ **Post date:** [March 31, 2014, 10:03am UTC](https://discuss.elastic.co/t/merge-segment-understanding/16679/3 "2014-03-31T10:03:08Z")
Thanks Binh. I'm curious about this because we're benchmarking our bulk indexing. And we've found out that the fastest bulk indexing strategy to be: - bulk indexing with 0 replica, no refresh, let ES do as little merge as possible - when indexing finished, optimize segments - replicates Is there some readings about the details/internals of lucene? We've the book Lucene in Action but it's mainly about core concepts and usage. 在 2014年3月28日星期五UTC+1下午8时32分46秒，Binh Ly写道： > The indexing buffer could also fill up which will flush to a segment. Also > the translog flush is not "exactly" deterministic, for example > "index.translog.interval" determines how often to check if the translog > needs to be flushed or not. Anyway, I wouldn't worry about it if I were > you. About the merge, I'd probably leave the defaults alone unless you are > absolutely sure changing them helps you. The more segments there are, the > more time it could take to do a merge. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/21034812-7c7e-4469-a3ad-7ceadde349e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/21034812-7c7e-4469-a3ad-7ceadde349e6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:39am UTC](https://discuss.elastic.co/t/merge-segment-understanding/16679/4 "2017-07-06T01:39:25Z")