# "now throttling indexing"
**URL:** https://discuss.elastic.co/t/now-throttling-indexing/22650
**Category:** Elasticsearch
**Created:** [March 13, 2015, 1:16am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650 "2015-03-13T01:16:32Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [March 13, 2015, 1:16am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/1 "2015-03-13T01:16:32Z")
I set `indices.store.throttle.type: none` in the elasticsearch.yml, and yet
this shows up in the logs:
now throttling indexing: numMergesInFlight=5, maxNumMerges=4
stop throttling indexing: numMergesInFlight=3, maxNumMerges=4
Did I misunderstand the purpose of this setting?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3b0f57e1-f78e-4782-ad4d-2dfe42bb5c17%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3b0f57e1-f78e-4782-ad4d-2dfe42bb5c17%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Alexander\_Jiteg](https://discuss.elastic.co/u/Alexander_Jiteg)
#### Post date: [March 13, 2015, 8:00am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/2 "2015-03-13T08:00:26Z")
I have the same problem.
On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote:
> I set `indices.store.throttle.type: none` in the elasticsearch.yml, and
> yet this shows up in the logs:
>
> now throttling indexing: numMergesInFlight=5, maxNumMerges=4
> stop throttling indexing: numMergesInFlight=3, maxNumMerges=4
>
> Did I misunderstand the purpose of this setting?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f1e4959b-1de9-45fe-ac9a-00c31268fb30%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f1e4959b-1de9-45fe-ac9a-00c31268fb30%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Alexander\_Jiteg](https://discuss.elastic.co/u/Alexander_Jiteg)
#### Post date: [March 13, 2015, 8:01am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/3 "2015-03-13T08:01:04Z")
Using ES 1.4.4 I should add.
On Friday, March 13, 2015 at 9:00:26 AM UTC+1, Alexander Jiteg wrote:
> I have the same problem.
>
> On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote:
>
> > I set `indices.store.throttle.type: none` in the elasticsearch.yml, and
> > yet this shows up in the logs:
> >
> > now throttling indexing: numMergesInFlight=5, maxNumMerges=4
> > stop throttling indexing: numMergesInFlight=3, maxNumMerges=4
> >
> > Did I misunderstand the purpose of this setting?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@mikemccand](https://discuss.elastic.co/u/mikemccand)
#### Post date: [March 13, 2015, 1:09pm UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/4 "2015-03-13T13:09:49Z")
That is the right setting to disable store throttling, but even without
throttling writes MB/sec for merges, the merges can still fall behind,
leading to index throttling. ES does this to protect the health of the
index because too many segments will cause all sorts of trouble.
What IO system is your index on? If you're on spinning disks you could try
setting index.merge.scheduler.max\_thread\_count to 1 since spinning disks
struggle with concurrent merges. See
> **[Merge | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-merge.html)**
Also, do you leave enough (at least 50%) free RAM to the OS for buffering
pages? This can make a difference with spinning disks since the OS has
more freedom to do read-ahead on the files being merged...
Mike McCandless
[http://www.elastic.co](http://www.elastic.co)
On Fri, Mar 13, 2015 at 1:01 AM, Alexander Jiteg \<
[alexander.jiteg@tickstar.com](mailto:alexander.jiteg@tickstar.com)\> wrote:
> Using ES 1.4.4 I should add.
>
> On Friday, March 13, 2015 at 9:00:26 AM UTC+1, Alexander Jiteg wrote:
>
> > I have the same problem.
> >
> > On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote:
> >
> > > I set `indices.store.throttle.type: none` in the elasticsearch.yml, and
> > > yet this shows up in the logs:
> > >
> > > now throttling indexing: numMergesInFlight=5, maxNumMerges=4
> > > stop throttling indexing: numMergesInFlight=3, maxNumMerges=4
> > >
> > > Did I misunderstand the purpose of this setting?
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKHUQPgPG9ovisOJ0q\_zQaZ-HoNK0pWsqrhC9r-z5gzBhoQ6JQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKHUQPgPG9ovisOJ0q_zQaZ-HoNK0pWsqrhC9r-z5gzBhoQ6JQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [March 13, 2015, 4:31pm UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/5 "2015-03-13T16:31:14Z")
On Fri, Mar 13, 2015 at 6:09 AM, Michael McCandless [mike@elastic.co](mailto:mike@elastic.co) wrote:
> That is the right setting to disable store throttling, but even without throttling writes MB/sec for merges, the merges can still fall behind, leading to index throttling. ES does this to protect the health of the index because too many segments will cause all sorts of trouble.
>
> What IO system is your index on? If you're on spinning disks you could try setting index.merge.scheduler.max\_thread\_count to 1 since spinning disks struggle with concurrent merges. See [Merge | Elasticsearch Guide [8.11] | Elastic](http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-merge.html)
>
> Also, do you leave enough (at least 50%) free RAM to the OS for buffering pages? This can make a difference with spinning disks since the OS has more freedom to do read-ahead on the files being merged...
The index is on an SSD (EC2 m3.large). I'll look into what else is
going on, had just noticed a few of those log entries, and thought
setting `indices.store.throttle.type: none` might be a quick fix 🙂
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHte5%2B%2BqrQzoQAicwz9Q%2BMprcNVhVqXWKGp\_CkKYefKNeWnO\_A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHte5%2B%2BqrQzoQAicwz9Q%2BMprcNVhVqXWKGp_CkKYefKNeWnO_A%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Igor\_Berman](https://discuss.elastic.co/u/Igor_Berman)
#### Post date: [September 2, 2015, 7:56am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/6 "2015-09-02T07:56:51Z")
Hi Erik,
have you resolved this issue somehow?
We have same problem with ES 1.6.0 and using SSDs but still throttling happening (which brings to big ratio of deleted docs)
have you understood where this maxNumMerges comes from?
thanks in advance
---
### Author:  [@mikemccand](https://discuss.elastic.co/u/mikemccand)
#### Post date: [September 2, 2015, 2:43pm UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/7 "2015-09-02T14:43:47Z")
You should disable store throttling when using SSDs: the default is horribly low (20 MB/sec) and very easily leads to merging falling behind, and then index throttling, under heavy indexing.
In ES 2.0 Lucene has switched to an adaptive IO throttle that sets itself according to merge backlog ...
---
### Author:  [@Igor\_Berman](https://discuss.elastic.co/u/Igor_Berman)
#### Post date: [September 2, 2015, 3:08pm UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/8 "2015-09-02T15:08:30Z")
Hi Michael,
actually we already disabled it with indices.store.throttle.type: none
still I see in HQ plugin that the rate of merges is around 8MB/s which is very low imho
we also increased index.merge.policy.reclaim\_deletes\_weight: 3
other settings:
index.load\_fixed\_bitset\_filters\_eagerly: false
index.refresh\_interval: 30s
index.translog.flush\_threshold\_size: 8gb
indices.cache.filter.size: 10%
indices.fielddata.cache.size: 10%
indices.memory.index\_buffer\_size: 30%
indices.store.throttle.type: none
threadpool.bulk.queue\_size: 3000
---
### Author:  [@mikemccand](https://discuss.elastic.co/u/mikemccand)
#### Post date: [September 2, 2015, 5:45pm UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/9 "2015-09-02T17:45:38Z")
Can you revert all settings to defaults and then see whether merges are still falling behind?
---
### Author:  [@Igor\_Berman](https://discuss.elastic.co/u/Igor_Berman)
#### Post date: [September 3, 2015, 8:04am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/10 "2015-09-03T08:04:07Z")
Hi Michael,
thanks for suggestions!
I'll try to talk to the team
---
### Author:  [@lisak](https://discuss.elastic.co/u/lisak)
#### Post date: [October 21, 2015, 6:09pm UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/11 "2015-10-21T18:09:52Z")
I have exactly the same problem, disabling throttling and refresh\_interval before bulk indexing doesn't help ... [Merge Thread busy even when refresh\_interval disabled for indexing](https://discuss.elastic.co/t/merge-thread-busy-even-when-refresh-interval-disabled-for-indexing/27657/7?u=lisak)
---
### Author:  [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)
#### Post date: [January 5, 2016, 10:05am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/12 "2016-01-05T10:05:34Z")
I had this problem, I am using elasticsearch 1.6.0 , and indices.store.throttle.type: none,index.refresh\_interval: 30s, index.translog.flush\_threshold\_size: 1gb, indices.memory.index\_buffer\_size: 20%, index.merge.scheduler.max\_thread\_count: 1. Is there some thing i could do to stop this, which can improve my indexing speed.
---
### Author:  [@mikemccand](https://discuss.elastic.co/u/mikemccand)
#### Post date: [January 5, 2016, 11:26am UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/13 "2016-01-05T11:26:45Z")
Try the suggestions here? [https://www.elastic.co/blog/performance-indexing-2-0](https://www.elastic.co/blog/performance-indexing-2-0)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:27pm UTC](https://discuss.elastic.co/t/now-throttling-indexing/22650/14 "2017-07-05T23:27:00Z")