# How to stop index from getting throttled? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 1, 2023, 8:11am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923 "2023-06-01T08:11:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 8:11am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/1 "2023-06-01T08:11:28Z")
Hi, I am ingesting netflow data into my ES 8.3.3 node at a very high rate. As I increase the ingest to ES where the index rate is about 30+K/s, I started to get the messages below: ```auto [INFO] [o.e.i.e.I.EngineMergeScheduler] ... now throttling indexing: numMergesInFlight=10, maxNumMerges=9 [INFO] [o.e.i.e.I.EngineMergeScheduler] ... stop throttling indexing: numMergesInFlight=8, maxNumMerges=9 ``` I have tried to increase the `refresh_interval` to 30s, but that did not help. I noticed that as I increase the ingest rate, I can never go above 34K/s index rate, and I would like to index at a much higher rate. There is a [StackOverflow post](https://stackoverflow.com/questions/29040039/how-to-prevent-elasticsearch-from-index-throttling) that talks about increasing `index.merge.scheduler.max_merge_count` and `index.merge.scheduler.max_thread_count`, but as the answer is a few years old, I'm not sure if it is still valid. I'm simply using the netflow module that came with filebeat and the default mapping. Would specifying the index mapping help (currently, most fields are by default mapped to both `text` and `keyword`, which is not necessary). What are some of the things I can do to prevent the index throttling and increase the index rate? Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2023, 8:15am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/2 "2023-06-01T08:15:24Z")
What is the size and specification of your cluster? What type of hardware are you using? Are you using local SSDs? ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 8:25am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/3 "2023-06-01T08:25:00Z")
I only have 1 node (`"number_of_replicas": "0"`). It is on a Linux server running RHEL 7.9, with 6TB of local SSDs, 755GB of RAM (64GB ringfenced for ES), and 96 CPUs. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2023, 8:29am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/4 "2023-06-01T08:29:59Z")
How many shards are you actively indexing into? How many concurrent indexing threads are you using in the process indexing data into Elasticsearch? What bulk size are you using? What is the average size of the documents you are indexing? ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 8:31am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/5 "2023-06-01T08:31:57Z")
I'm only indexing into 1 shard (index). How do I find out how many concurrent indexing threads and bulk size I'm using? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2023, 8:32am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/6 "2023-06-01T08:32:48Z")
That depends on what you are using to index data into Elasticsearch, e.g. Logstash or one of the beats. ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 8:39am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/7 "2023-06-01T08:39:06Z")
I'm using filebeat. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2023, 8:40am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/8 "2023-06-01T08:40:39Z")
How is Filebeat configured? ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 8:44am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/9 "2023-06-01T08:44:45Z")
I configured the following in my `filebeat.yml`. ```auto output.elasticsearch: bulk_max_size: 4000 worker: 8 ``` In `netflow.yml`, I also have `queue_size: 64000`. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2023, 8:50am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/10 "2023-06-01T08:50:46Z")
I have not tuned Filebeat in a very long time so may have to leave that for someone else. Have you tried increasing the number of workers or the bulk size? Did that have any effect? ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 8:55am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/11 "2023-06-01T08:55:39Z")
I have yet to tune these parameters, as I did not think that the problem might lie with filebeat. I'll try increasing their values and see if it works. Thanks! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2023, 8:57am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/12 "2023-06-01T08:57:13Z")
What type of SSD are you using? Am surprised to see indexing throttled due to merging when using SSDs. ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 9:04am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/13 "2023-06-01T09:04:14Z")
They're SATA SSD, each 960GB. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2023, 9:12am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/14 "2023-06-01T09:12:39Z")
Can you try increasing the number of primary shards to 2 for the next active index and see if that makes any difference? ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 1, 2023, 9:33am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/15 "2023-06-01T09:33:27Z")
Ok, I'll give that a try. ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 6, 2023, 6:00am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/16 "2023-06-06T06:00:55Z")
Increasing the number of primary shards to 2 seems to help. Previously, I would get a lot of "throttling indexing" messages when the indexing rate goes above 30K/s. Now, it was only after 3 hours that I got a "throttling indexing" message (indexing rate consistently above 30K/s), then another hour before getting another 2 such messages. I was even able to reach 35K/s without getting the message. I'm not sure if the size of the index matters too? As the index was a custom index, I have not been able to apply the ILM on it yet. So the last "throttling indexing" message occurred when the index is 480GB containing 490M documents. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 6, 2023, 6:04am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/17 "2023-06-06T06:04:20Z")
I'd reduce the `bulk_max_size` to 3000, I think the default if 2000 so doubling it might not be the first best step. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 6, 2023, 6:19am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/18 "2023-06-06T06:19:09Z")
> [@hjazz6](#): > > I'm not sure if the size of the index matters too? As the index was a custom index, I have not been able to apply the ILM on it yet. So the last "throttling indexing" message occurred when the index is 480GB containing 490M documents. That could very well explain it. I assumed you were using time based indices. You will not be able to apply ILM to an existing index, so I would recommend setting up a new data stream with ILM and direct all new data there. Once the existing index only contains old data I would remove it manually. ---
**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\ **Post date:** [June 7, 2023, 3:26am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/19 "2023-06-07T03:26:32Z")
I've applied ILM on the index so that it rollovers every 50GB, and I have not seen the "throttling indexing" message anymore. Thanks for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2023, 3:26am UTC](https://discuss.elastic.co/t/how-to-stop-index-from-getting-throttled/334923/20 "2023-07-05T03:26:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.