# Elasticsearch 7.17.10 indexing bottleneck on i3.2xlarge and d3.2xlarge nodes in EKS **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2023, 8:09pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503 "2023-05-15T20:09:54Z")\ **Posts on this page:** 14\ **Page:** 3
**Author:** ![Chris\_Austin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_austin/32/121123_2.png) [@Chris\_Austin](https://discuss.elastic.co/u/Chris_Austin)\ **Post date:** [May 23, 2023, 5:42pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/41 "2023-05-23T17:42:50Z")
> [@DavidTurner](#): > > But are all nodes now queueing up writes or is the queue still concentrated on a subset of the nodes? Still pretty concentrated. Indexing rate by node, 7d: ![Screen Shot 2023-05-23 at 12.39.48](https://us1.discourse-cdn.com/elastic/original/3X/6/5/65b459ef10e46b7b48ff6b3d0889ddbbc0f70e29.jpeg) Write queue by node, 7d: ![Screen Shot 2023-05-23 at 12.40.31](https://us1.discourse-cdn.com/elastic/original/3X/3/3/337146bfbeb6379ca847b04923c72b01f8f1e448.png) Write queue by node, 2d to see more recent data. ![Screen Shot 2023-05-23 at 12.41.21](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9d51fca1947476f44bd3fa81f8f939f7d106f92.jpeg) ---
**Author:** ![Chris\_Austin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_austin/32/121123_2.png) [@Chris\_Austin](https://discuss.elastic.co/u/Chris_Austin)\ **Post date:** [May 23, 2023, 5:48pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/42 "2023-05-23T17:48:23Z")
> [@DavidTurner](#): > > Could you share the output of `GET _nodes/data_hot:true/hot_threads?threads=9999` while the cluster is trying to process a backlog? The output is too large for a comment - preferred way to provide this? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 23, 2023, 6:07pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/43 "2023-05-23T18:07:34Z")
> [@Chris\_Austin](#): > > Still pretty concentrated. Any idea why? Are the hotter nodes holding more shards seeing heavy indexing? There's still a lot of empty queues here. > [@Chris\_Austin](#): > > The output is too large for a comment - preferred way to provide this? Can you use [https://gist.github.com](https://gist.github.com)? ---
**Author:** ![Chris\_Austin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_austin/32/121123_2.png) [@Chris\_Austin](https://discuss.elastic.co/u/Chris_Austin)\ **Post date:** [May 23, 2023, 6:13pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/44 "2023-05-23T18:13:21Z")
> ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 23, 2023, 6:25pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/45 "2023-05-23T18:25:45Z")
Thanks. So again the hot threads indicates most nodes are underutilised, with many idle `write` threads. There's a few over-hot nodes that could be a bottleneck tho, and `alerts-es-hot-8` looks to be struggling a bit with IO, spending a lot of time in `force0` (i.e. `fsync`): ```auto $ cat es-tasks| grep -B2 force0 100.0% [cpu=38.2%, other=61.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[alerts-es-hot-8][write][T#8]' 5/10 snapshots sharing following 27 elements java.base@20.0.1/sun.nio.ch.UnixFileDispatcherImpl.force0(Native Method) -- 100.0% [cpu=33.3%, other=66.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[alerts-es-hot-8][write][T#6]' 5/10 snapshots sharing following 27 elements java.base@20.0.1/sun.nio.ch.UnixFileDispatcherImpl.force0(Native Method) -- java.base@20.0.1/java.lang.Thread.run(Thread.java:1623) unique snapshot java.base@20.0.1/sun.nio.ch.UnixFileDispatcherImpl.force0(Native Method) -- 100.0% [cpu=22.9%, other=77.1%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[alerts-es-hot-8][write][T#1]' 5/10 snapshots sharing following 27 elements java.base@20.0.1/sun.nio.ch.UnixFileDispatcherImpl.force0(Native Method) ``` I'd suggest focussing on why there's such an imbalance in work across your nodes. ---
**Author:** ![Chris\_Austin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_austin/32/121123_2.png) [@Chris\_Austin](https://discuss.elastic.co/u/Chris_Austin)\ **Post date:** [May 23, 2023, 6:33pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/46 "2023-05-23T18:33:37Z")
> [@DavidTurner](#): > > Also I don't think you ever looked at TCP buffer utilisation which would help to show if the holdup was either in the network or on the ES transport threads: Try `ss -tonie` on all the hot nodes and look for connections with big buffers. > ---
**Author:** ![Chris\_Austin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_austin/32/121123_2.png) [@Chris\_Austin](https://discuss.elastic.co/u/Chris_Austin)\ **Post date:** [May 23, 2023, 6:36pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/47 "2023-05-23T18:36:12Z")
> [@DavidTurner](#): > > I'd suggest focussing on why there's such an imbalance in work across your nodes. Is there a recommended way to do this with 7.17.10? I thought that shard allocation based on size or write only became available in 8. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 23, 2023, 7:21pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/48 "2023-05-23T19:21:42Z")
Thanks, ok, no significant holdups on the network layer it seems. > [@Chris\_Austin](#): > > Is there a recommended way to do this with 7.17.10? I thought that shard allocation based on size or write only became available in 8. Firstly just work out if it's needed (using e.g. `GET _cat/shards`). If there is a hot spot for a particular index then sometimes it's because the shards don't divide equally across the available nodes so it can be addressed just by changing the number of shards/replicas. Otherwise you can use the `index.routing.allocation.total_shards_per_node` index setting to force the shards of particular indices to spread out. ---
**Author:** ![Chris\_Austin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_austin/32/121123_2.png) [@Chris\_Austin](https://discuss.elastic.co/u/Chris_Austin)\ **Post date:** [May 23, 2023, 8:07pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/49 "2023-05-23T20:07:45Z")
> [@DavidTurner](#): > > Firstly just work out if it's needed (using e.g. `GET _cat/shards`). If there is a hot spot for a particular index then sometimes it's because the shards don't divide equally across the available nodes so it can be addressed just by changing the number of shards/replicas. There are 217 empty indices that were created prematurely, but I've tested removing them without any noticeable impact to throughput - I was hoping that the shard rebalancing would have fewer "lucky" nodes that got mostly-idle shards. I also used the API to move busy shards from the busy nodes to the nodes with the lowest cpu usage, but that's obviously not a sustainable practice unless I write a service to redistribute them for me. I'd like to move to ES 8, which might help, but I'm getting some pressure to consider using opensearch to offload that troubleshooting effort, and upgrading past 7 makes that path a bit harder. I'd prefer to stay with ES, so thank you again for all your help in this thread - it's very appreciated. > [@DavidTurner](#): > > Otherwise you can use the `index.routing.allocation.total_shards_per_node` index setting to force the shards of particular indices to spread out. I believe this was addressed in [this part of the thread](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/26) - only one index would be affected, and I've dropped that one from 5 shards to 4, which should allow it to be distributed across all nodes without any overlap. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 23, 2023, 8:07pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/50 "2023-05-23T20:07:45Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 23, 2023, 8:47pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/51 "2023-05-23T20:47:16Z")
> [@Chris\_Austin](#): > > I'm getting some pressure to consider using opensearch to offload that troubleshooting effort, AFAIK OpenSearch uses the same shard allocation algorithm as Elasticsearch 7.x, so I don't think that will change anything. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [May 24, 2023, 11:22am UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/52 "2023-05-24T11:22:49Z")
Since you're managing this Elasticsearch cluster via ECK, could you provide the Elasticsearch YAML file that you're using to deploy your cluster? ---
**Author:** ![Chris\_Austin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_austin/32/121123_2.png) [@Chris\_Austin](https://discuss.elastic.co/u/Chris_Austin)\ **Post date:** [May 25, 2023, 1:03pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/53 "2023-05-25T13:03:25Z")
> [@DavidTurner](#): > > AFAIK OpenSearch uses the same shard allocation algorithm as Elasticsearch 7.x, so I don't think that will change anything. Yeah, the opensearch piece is just to make it someone else's problem with a support contract. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2023, 1:03pm UTC](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503/54 "2023-06-22T13:03:37Z")
Closed on June 22, 2023, 1:03pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/elasticsearch-7-17-10-indexing-bottleneck-on-i3-2xlarge-and-d3-2xlarge-nodes-in-eks/333503.md?page=2)