Urgent Help Needed Elastic Shutting down ingestion & All the write load is being transferred to one of the hot nodes

Elastic Stack Elasticsearch
1

I have issue with cluster all the time i can see is queue on one of the hot nodes at a time.

node_name name active queue rejected
elastic-frozen2 write 0 0 0
elastic-hot1 write 2 0 0
elastic-hot5 write 32 3230 0
elastic-master1 write 0 0 0
elastic-hot6 write 13 0 0
elastic-master3 write 0 0 0
elastic-coordinating1 write 0 0 0
elastic-coordinating2 write 0 0 0
elastic-ml1 write 0 0 0
elastic-frozen1 write 0 0 0
elastic-hot4 write 25 0 0
elastic-hot3 write 5 0 0
elastic-ingest2 write 0 0 0
elastic-master2 write 0 0 0
elastic-ingest1 write 0 0 0
elastic-hot2 write 32 1 0
elastic-ingest3 write 0 0 0
elastic-ml2 write 0 0 0

it is going to sometime on of hot node high not equally diving it.

node_name name active queue rejected
elastic-hot5 write 32 4505 0
elastic-hot6 generic 12 0 0
elastic-hot4 generic 12 0 0
elastic-hot5 generic 10 0 0
elastic-hot6 merge 9 0 0
elastic-hot2 generic 6 0 0
elastic-hot2 merge 6 0 0
elastic-hot3 merge 5 0 0
elastic-hot5 merge 4 0 0
elastic-hot1 generic 2 0 0
elastic-hot1 management 2 0 0
elastic-hot1 merge 2 0 0
elastic-hot5 flush 2 0 0
elastic-coordinating1 management 2 0 0
elastic-hot4 merge 2 0 0
elastic-hot3 generic 2 0 0
elastic-master2 inference_utility 2 0 0
elastic-hot2 flush 2 0 0
elastic-frozen2 inference_utility 1 0 0
elastic-frozen2 management 1 0 0
elastic-hot1 inference_utility 1 0 0
elastic-hot1 refresh 1 0 0
elastic-hot1 system_write 1 0 0
elastic-hot1 watcher 1 0 0
elastic-hot5 inference_utility 1 0 0
elastic-hot5 management 1 0 0
elastic-hot5 refresh 1 0 0
elastic-master1 inference_utility 1 0 0
elastic-master1 management 1 0 0
elastic-hot6 flush 1 0 0
elastic-hot6 inference_utility 1 0 0
elastic-hot6 management 1 0 0
elastic-hot6 write 1 0 0
elastic-master3 inference_utility 1 0 0
elastic-master3 management 1 0 0
elastic-coordinating1 inference_utility 1 0 0
elastic-coordinating2 inference_utility 1 0 0
elastic-coordinating2 management 1 0 0
elastic-ml1 inference_utility 1 0 0
elastic-ml1 management 1 0 0
elastic-frozen1 inference_utility 1 0 0
elastic-frozen1 management 1 0 0
elastic-hot4 inference_utility 1 0 0
elastic-hot4 management 1 0 0
elastic-hot3 inference_utility 1 0 0
elastic-hot3 management 1 0 0
elastic-ingest2 inference_utility 1 0 0
elastic-ingest2 management 1 0 0
elastic-master2 management 1 0 0
elastic-ingest1 inference_utility 1 0 0
elastic-ingest1 management 1 0 0
elastic-ingest1 write_coordination 1 0 0
elastic-hot2 inference_utility 1 0 0
elastic-hot2 management 1 0 0
elastic-ingest3 inference_utility 1 0 0
elastic-ingest3 management 1 0 0
elastic-ml2 inference_utility 1 0 0
elastic-ml2 management 1 0 0
elastic-frozen2 analyze 0 0 0

2

Hi Kishor,

Shutting down the ingest nodes will not redistribute the indexing load across the hot nodes. Ingest nodes process the pipelines, but the final write is handled by the hot data nodes that contain the target shards.

In your output, elastic-hot5 has all 32 write threads active and thousands of queued operations. The fact that rejected is still 0 only means the queue has not reached its limit yet.

This looks like write hot spotting, usually related to shard allocation, too few primary shards in the active index, custom routing, or a resource difference on that node.

Could you share the Elasticsearch version and the output of:

GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors

GET _cat/allocation?v&s=node

GET _cat/shards/?v&s=index,shard,prirep,state,node

GET _nodes/elastic-hot5/hot_threads

I would not increase the write queue as the first action. The priority is to understand why the write workload or processing time is concentrated on hot5

Hotspotting

3 
GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors

Summary

name                   node.role cpu heap.percent disk.used_percent load_1m 
elastic-coordinating1 -           1           48              6.82    0.08 
elastic-coordinating2 -           1           47              6.83    0.25 elastic-frozen1       f           1           57             86.17    0.19 elastic-frozen2       f           1           25             86.16    0.22 elastic-hot1          his        36           67             69.45   13.78 elastic-hot2          his        46           27             66.09   21.09 elastic-hot3          his        12           44             67.91    7.82 elastic-hot4          his        11           16             72.62    6.71 elastic-hot5          his         5           23             65.01    2.98 elastic-hot6          hs         14           61             73.09    7.24 elastic-ingest1       i           3           64              7.58    0.79 elastic-ingest2       i           3           59              7.32    0.75 elastic-ingest3       i           4           15              7.59    0.76 elastic-master1       m           7           11              7.11    0.13 elastic-master2       m          14           30              7.07    1.00 elastic-master3       m           1           50              6.92    0.00 elastic-ml1           lt          0           28              0.00    0.00 elastic-ml2           lt          0            9              0.00    0.08


GET _cat/allocation?v&s=node&h=node,node.node.role,shards,shards.undesired,write_load.forecast,disk.indices.forecast,disk.indices,disk.used,disk.avail,disk.total,disk.percent

Summary

node                         shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent
elastic-frozen1    533                0                 0.0                 7.3tb           0b     3.4tb    561.7gb      3.9tb           86
elastic-frozen2    123                0                 0.0                 3.5tb           0b     3.4tb    562.1gb      3.9tb           86
elastic-hot1       544              325   16.37320425123251                 5.7tb        5.5tb     5.5tb      2.4tb      7.9tb           69
elastic-hot2       316                0  23.767974006518322                 5.5tb        5.2tb     5.3tb      2.6tb      7.9tb           66
elastic-hot3       438               30   9.503747177041182                 5.6tb        5.3tb     5.4tb      2.5tb      7.9tb           67
elastic-hot4       714              242   4.254057560988934                 5.9tb        5.7tb     5.8tb      2.1tb      7.9tb           72
elastic-hot5       278               33   4.385566225164219                 5.3tb        5.1tb     5.2tb      2.7tb      7.9tb           65
elastic-hot6       892              397    4.47431549184445                 5.9tb        5.7tb     5.8tb      2.1tb      7.9tb           72
4

Thanks for sharing this.

The interesting part is that hot5 is not the busiest node in this snapshot. Hot2 has higher CPU, load and forecasted write load, so the write pressure seems to move between nodes.

That usually points more to the location of the active primary shards than to a problem with one specific node.

I would check two things:

  1. which node holds the primary shards of the index receiving the writes
  2. whether the number of undesired shards is going down over time

You currently have more than 1,000 undesired shards, so Elasticsearch is still trying to improve the balance.

When the queue increases again, capture the shard distribution for the target index at the same time. That should make it easier to confirm whether one primary shard is receiving most of the traffic.

I would not change the thread pool or manually move shards yet.

5

i have increased the number of primaries to 3 for few data-streams which were having highest ingestion.

As current settings are to keep data for 7 days in cluster only for 6.5 tb per day primary data with 1 replica and have many small shards like mbs only. where to keep the policy was applied to rollover per day.

but as smaller shards are increasing too much that was i think also impacting the cluster as well.

so increasing the number of primaries has given something

node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent
elastic-frozen1 521 0 0.0 7.4tb 0b 3.4tb 561.7gb 3.9tb 86
elastic-frozen2 136 0 0.0 3.9tb 0b 3.4tb 562gb 3.9tb 86
elastic-hot1 692 122 13.963806701068636 5.5tb 5.2tb 5.3tb 2.6tb 7.9tb 66
elastic-hot2 566 61 8.727707970013126 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65
elastic-hot3 235 9 14.113464883152382 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65
elastic-hot4 763 128 7.9550333410730065 5.5tb 5.2tb 5.3tb 2.6tb 7.9tb 66
elastic-hot5 640 36 9.707710075522193 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65
elastic-hot6 250 9 8.160587552403719 5.4tb 5.2tb 5.2tb 2.7tb 7.9tb 66

what would you suggest and also i have increased the concurent_shard Balancing to 4 and recoveries as well to 4. also identifying the smaller shards and making to hold till the meat the size.

one more thing i have ingest role in the hot nodes as i have the dedicated ingest nodes, but when i wanted to remove the ingest role from all the apm ingestion stopped.

fleet >settings > out has ingest nodes ip instead of coordinate or hot nods.

but i have seen which nodes should be there as per elk suggestion in this scanario given below, as i need CPU i know and also we are going to have CCS, ML

name ram.max disk.total disk.avail disk.used node.role
elastic-frozen2 31.3gb 3.9tb 562gb 3.4tb f
elastic-hot1 62.7gb 7.9tb 2.6tb 5.3tb his
elastic-hot5 62.7gb 7.9tb 2.7tb 5.2tb his
elastic-master1 31.3gb 193.6gb 179.8gb 13.7gb m
elastic-hot6 62.7gb 7.9tb 2.6tb 5.3tb hs
elastic-master3 31.3gb 193.6gb 180.3gb 13.2gb m
elastic-coordinating1 15.6gb 193.6gb 180.4gb 13.2gb -
elastic-coordinating2 15.6gb 193.6gb 180.4gb 13.2gb -
elastic-ml1 31.3gb 8192pb 8191.9pb 16.2tb lt
elastic-frozen1 31.3gb 3.9tb 561.7gb 3.4tb f
elastic-hot4 62.7gb 7.9tb 2.6tb 5.3tb his
elastic-hot3 62.7gb 7.9tb 2.7tb 5.2tb his
elastic-ingest2 31.3gb 193.6gb 179.4gb 14.2gb i
elastic-master2 31.3gb 193.6gb 179.9gb 13.6gb m
elastic-ingest1 62.7gb 193.6gb 178.9gb 14.6gb i
elastic-hot2 62.7gb 7.9tb 2.7tb 5.2tb his
elastic-ingest3 31.3gb 193.6gb 178.9gb 14.7gb i
elastic-ml2 31.3gb 8192pb 8191.9pb 16.2tb lt

6

maybe me, but it seems you didn't attach output from:

GET _cat/shards/?v&s=index,shard,prirep,state,node

or

GET _nodes/elastic-hot5/hot_threads

Also, the formatting of the output on /_cat/nodes was a bit messed up on my browser at least, here's it tidied it up:

name                   node.role cpu heap.percent disk.used_percent load_1m
elastic-coordinating1 -           1           48              6.82    0.08
elastic-coordinating2 -           1           47              6.83    0.25
elastic-frozen1       f           1           57             86.17    0.19
elastic-frozen2       f           1           25             86.16    0.22
elastic-hot1          his        36           67             69.45   13.78
elastic-hot2          his        46           27             66.09   21.09
elastic-hot3          his        12           44             67.91    7.82
elastic-hot4          his        11           16             72.62    6.71
elastic-hot5          his         5           23             65.01    2.98
elastic-hot6          hs         14           61             73.09    7.24
elastic-ingest1       i           3           64              7.58    0.79
elastic-ingest2       i           3           59              7.32    0.75
elastic-ingest3       i           4           15              7.59    0.76
elastic-master1       m           7           11              7.11    0.13
elastic-master2       m          14           30              7.07    1.00
elastic-master3       m           1           50              6.92    0.00
elastic-ml1           lt          0           28              0.00    0.00
elastic-ml2           lt          0            9              0.00    0.08

Did you change something in this cluster recently, eg add nodes, or have some cluster issues, like nodes leaving/joining, as the value of shards.undesired on some of the hot nodes is a bit surprising.

What version are you running? There has been a few threads about clusters getting into a seemingly endless cycle of re-balancing shards.