# Elasticsearch indexing performance: throttle merging
**URL:** https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549
**Category:** Elasticsearch
**Created:** [January 13, 2017, 5:14pm UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549 "2017-01-13T17:14:14Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@antonbormotov](https://discuss.elastic.co/u/antonbormotov)
#### Post date: [January 13, 2017, 5:14pm UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/1 "2017-01-13T17:14:14Z")
We are importing data to elasticsearch cluster in few indices, around `~10gb` each.
At the same time, we care about search on existing indices, few of them are small-`~100mb`, few of them are big-`~10gb`.
In order to optimize indexing, we:
- use `bulk` api with optimized bulk size;
- set refresh interval to `-1`;
- set replication factor to `0`;
Now, we are trying to understand how merge throttling can help. How search and segment merging are related, if search only against existing indices?
According to this [article](https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html#segments-and-merging), we can disable merge throttling.
- Does that mean merges will "eat" disks i/o?
- Does that mean merges won't happen at all and we have to `_forcemerge` manually, after indexing is done? Should be worried about max open file descriptors in such case?
According to these [article](https://www.elastic.co/blog/performance-indexing-2-0) and [pull request](https://github.com/elastic/elasticsearch/pull/9243/files) we shouldn't touch merging settings at all.
Very confused here, any help is highly appreciated.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 16, 2017, 12:39am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/2 "2017-01-16T00:39:04Z")
Don't worry about it, let ES handle the merging automatically 🙂
Your initial 3 steps are all you need to do!
---
### Author:  [@antonbormotov](https://discuss.elastic.co/u/antonbormotov)
#### Post date: [January 16, 2017, 2:29am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/3 "2017-01-16T02:29:04Z")
@warkolm would be grateful, if you can add more details and answer 2 questions above. I want to understand how does it work and what actually happens.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 16, 2017, 2:47am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/4 "2017-01-16T02:47:23Z")
Which two questions?
---
### Author:  [@antonbormotov](https://discuss.elastic.co/u/antonbormotov)
#### Post date: [January 16, 2017, 2:50am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/5 "2017-01-16T02:50:26Z")
> [@antonbormotov](#):
>
> - Does that mean merges will "eat" disks i/o?
> - Does that mean merges won't happen at all and we have to \_forcemerge manually, after indexing is done? Should be worried about max open file descriptors in such case?
These two, Mark.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 16, 2017, 2:54am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/6 "2017-01-16T02:54:12Z")
The best answer to those is don't disable merging as I mentioned.
Otherwise yes merges use IO, if you disable them then they won't happen and a force merge is required.
---
### Author:  [@antonbormotov](https://discuss.elastic.co/u/antonbormotov)
#### Post date: [January 16, 2017, 6:47am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/7 "2017-01-16T06:47:11Z")
@warkolm, sorry for molestation, documentation is really poor regarding this internal logic.
As far as I understood, importing data at certain rate might cause merging processes 'eat' all available disk i/o.
In order to keep some room for search queries, there is configuration `indices.store.throttle.max_bytes_per_sec` that throttles indexing threads if merging rate is higher than this number.
Using configuration option `indices.store.throttle.type` we can disable/enable index throttling.
Looks like `merge throttling` actually means `index throttling`.
See pr [here](https://github.com/elastic/elasticsearch/issues/6066) and qbox article [here](https://qbox.io/blog/a-z-guide-on-scaling-elasticsearch).
I thought if merges won't happen, it might across max open file descriptors number in OS, if index is huge.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 16, 2017, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/8 "2017-01-16T08:15:18Z")
Are you indexing into these indices continuously or doing bulk inserts/updates periodically?
---
### Author:  [@antonbormotov](https://discuss.elastic.co/u/antonbormotov)
#### Post date: [January 16, 2017, 11:01am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/9 "2017-01-16T11:01:05Z")
Periodically, daily basis, new indices every day.
---
### Author:  [@mikemccand](https://discuss.elastic.co/u/mikemccand)
#### Post date: [January 16, 2017, 12:23pm UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/10 "2017-01-16T12:23:15Z")
Since ES 2.x, the IO throttling is handled automatically by Lucene, meaning it starts at 20 MB/sec throttle on writing bytes to the merged segment. It then increases that rate when merges fall behind, and decreases it otherwise. This means the merges, over time, only soak up as much IO bandwidth as is needed to keep up with your rate of indexing.
You don't need to `forceMerge` yourself: the merges will happen naturally as you are indexing.
Mike McCandless
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 13, 2017, 12:23pm UTC](https://discuss.elastic.co/t/elasticsearch-indexing-performance-throttle-merging/71549/11 "2017-02-13T12:23:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.