"now throttling indexing"

Elastic Stack Elasticsearch
1

I set indices.store.throttle.type: none in the elasticsearch.yml, and yet
this shows up in the logs:

now throttling indexing: numMergesInFlight=5, maxNumMerges=4
stop throttling indexing: numMergesInFlight=3, maxNumMerges=4

Did I misunderstand the purpose of this setting?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3b0f57e1-f78e-4782-ad4d-2dfe42bb5c17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I have the same problem.

On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote:

I set indices.store.throttle.type: none in the elasticsearch.yml, and
yet this shows up in the logs:

now throttling indexing: numMergesInFlight=5, maxNumMerges=4
stop throttling indexing: numMergesInFlight=3, maxNumMerges=4

Did I misunderstand the purpose of this setting?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f1e4959b-1de9-45fe-ac9a-00c31268fb30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Using ES 1.4.4 I should add.

On Friday, March 13, 2015 at 9:00:26 AM UTC+1, Alexander Jiteg wrote:

I have the same problem.

On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote:

I set indices.store.throttle.type: none in the elasticsearch.yml, and
yet this shows up in the logs:

now throttling indexing: numMergesInFlight=5, maxNumMerges=4
stop throttling indexing: numMergesInFlight=3, maxNumMerges=4

Did I misunderstand the purpose of this setting?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

That is the right setting to disable store throttling, but even without
throttling writes MB/sec for merges, the merges can still fall behind,
leading to index throttling. ES does this to protect the health of the
index because too many segments will cause all sorts of trouble.

What IO system is your index on? If you're on spinning disks you could try
setting index.merge.scheduler.max_thread_count to 1 since spinning disks
struggle with concurrent merges. See

Also, do you leave enough (at least 50%) free RAM to the OS for buffering
pages? This can make a difference with spinning disks since the OS has
more freedom to do read-ahead on the files being merged...

Mike McCandless

http://www.elastic.co

On Fri, Mar 13, 2015 at 1:01 AM, Alexander Jiteg <
alexander.jiteg@tickstar.com> wrote:

Using ES 1.4.4 I should add.

On Friday, March 13, 2015 at 9:00:26 AM UTC+1, Alexander Jiteg wrote:

I have the same problem.

On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote:

I set indices.store.throttle.type: none in the elasticsearch.yml, and
yet this shows up in the logs:

now throttling indexing: numMergesInFlight=5, maxNumMerges=4
stop throttling indexing: numMergesInFlight=3, maxNumMerges=4

Did I misunderstand the purpose of this setting?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKHUQPgPG9ovisOJ0q_zQaZ-HoNK0pWsqrhC9r-z5gzBhoQ6JQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

On Fri, Mar 13, 2015 at 6:09 AM, Michael McCandless mike@elastic.co wrote:

That is the right setting to disable store throttling, but even without throttling writes MB/sec for merges, the merges can still fall behind, leading to index throttling. ES does this to protect the health of the index because too many segments will cause all sorts of trouble.

What IO system is your index on? If you're on spinning disks you could try setting index.merge.scheduler.max_thread_count to 1 since spinning disks struggle with concurrent merges. See Merge | Elasticsearch Guide [8.11] | Elastic

Also, do you leave enough (at least 50%) free RAM to the OS for buffering pages? This can make a difference with spinning disks since the OS has more freedom to do read-ahead on the files being merged...

The index is on an SSD (EC2 m3.large). I'll look into what else is
going on, had just noticed a few of those log entries, and thought
setting indices.store.throttle.type: none might be a quick fix :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHte5%2B%2BqrQzoQAicwz9Q%2BMprcNVhVqXWKGp_CkKYefKNeWnO_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

Hi Erik,
have you resolved this issue somehow?
We have same problem with ES 1.6.0 and using SSDs but still throttling happening (which brings to big ratio of deleted docs)
have you understood where this maxNumMerges comes from?
thanks in advance

7

You should disable store throttling when using SSDs: the default is horribly low (20 MB/sec) and very easily leads to merging falling behind, and then index throttling, under heavy indexing.

In ES 2.0 Lucene has switched to an adaptive IO throttle that sets itself according to merge backlog ...

8

Hi Michael,

actually we already disabled it with indices.store.throttle.type: none
still I see in HQ plugin that the rate of merges is around 8MB/s which is very low imho
we also increased index.merge.policy.reclaim_deletes_weight: 3

other settings:
index.load_fixed_bitset_filters_eagerly: false
index.refresh_interval: 30s
index.translog.flush_threshold_size: 8gb
indices.cache.filter.size: 10%
indices.fielddata.cache.size: 10%
indices.memory.index_buffer_size: 30%
indices.store.throttle.type: none
threadpool.bulk.queue_size: 3000

9

Can you revert all settings to defaults and then see whether merges are still falling behind?

10

Hi Michael,
thanks for suggestions!
I'll try to talk to the team

11

I have exactly the same problem, disabling throttling and refresh_interval before bulk indexing doesn't help ... Merge Thread busy even when refresh_interval disabled for indexing

12

I had this problem, I am using elasticsearch 1.6.0 , and indices.store.throttle.type: none,index.refresh_interval: 30s, index.translog.flush_threshold_size: 1gb, indices.memory.index_buffer_size: 20%, index.merge.scheduler.max_thread_count: 1. Is there some thing i could do to stop this, which can improve my indexing speed.

13

Try the suggestions here? https://www.elastic.co/blog/performance-indexing-2-0