Slow queries - is ES waiting for resources?

Elastic Stack Elasticsearch
1

Hello,

I run the simple query with one term filter. During the heavy indexing into ES, the search took:

  • for 40 shards - 0.2s
  • for 80 shards - >10s

My elastic setup:
Elastic 8.5
Node: 5 x 16vCPU, 64GB

  • 5 data nodes (15 vcpu, 48GB ram)
  • 3 elastic-master
  • 5 elastic-client

In the Kibana profiler, for both cases, every step has a similar time. The difference is in the Cumulative time.

In metrics I don’t observe a slow queries:

image
image2000×355 127 KB

Why do we observe a slow query on 80 shards? Which metric should I use to detect problem? (I am thinking that ES is waiting for connection or thread)

Patryk.

2

One of the most common limiting factors for Elasticsearch performance is disk I/O. What type of storage are you using? What does disk utilisation and await look like during the time the query runs?

3

We use Azure Premium SSD:

StorageClass:  managed-csi-premium
Capacity:      3Ti

image
image2560×293 45.2 KB

await look like during the time the query runs?

How can I check it?

4

I would run iostat -x on one of the data nodes.