# Query intermittent performance issue

**URL:** <https://discuss.elastic.co/t/query-intermittent-performance-issue/358473>\
**Category:** Elasticsearch\
**Created:** [April 30, 2024, 9:49am UTC](https://discuss.elastic.co/t/query-intermittent-performance-issue/358473 "2024-04-30T09:49:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul6](https://avatars.discourse-cdn.com/v4/letter/p/ee7513/32.png) [@Paul6](https://discuss.elastic.co/u/Paul6)\
**Post date:** [April 30, 2024, 9:49am UTC](https://discuss.elastic.co/t/query-intermittent-performance-issue/358473/1 "2024-04-30T09:49:31Z")

</div>

Hello,

I’m on Elastic Cloud 8.12.

I’m facing an issue where the same query can take from 500ms to more than 30s.

We are working with rather simple data stored in data streams but we also reproduced this behavior with classic Elasticsearch indices.

I have profile the query and successfully reproduced similar issues with very simple queries such as a term filter or a range filter: { "query": {"bool": {"filter": [{ "range": {"\_expirationDate": { "gte": "2024-04-10T14:11:40Z" } } }]} }, "size": 501}.  
For instance, about 12s spent just for this range query on a single index (all spent in the match section of the profiler).

If I run the same query directly after, the performance will be much better thanks to the cache.

I also investigated Elasticsearch metrics in Kibana and CPU is always below 30%, JVM Head memory is fine, but there are spikes in the read I/O that might explain the issue:

![ioRead](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0e82d08fd0d7ab822a94728a1d4ed3913b196368.png)

I suspect a configuration issue, maybe on the indexes shards or the lack of OS cache memory but I’m fairly new to ES and not sure on how to continue my investigations.

Thanks,  
Paul

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 30, 2024, 10:08am UTC](https://discuss.elastic.co/t/query-intermittent-performance-issue/358473/2 "2024-04-30T10:08:46Z")

</div>

Bonjour Paul 😉

What is the output of:

```auto
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

```

What is the configuration you chose for the cloud instance? What kind of "hardware profile" and how much memory?

Note that I suspect as well that if you remove:

```
"size": 501

```

from the request, that could be faster? Could you check that?

---

<div class="post-metadata">

**Author:** ![Paul6](https://avatars.discourse-cdn.com/v4/letter/p/ee7513/32.png) [@Paul6](https://discuss.elastic.co/u/Paul6)\
**Post date:** [April 30, 2024, 11:22am UTC](https://discuss.elastic.co/t/query-intermittent-performance-issue/358473/3 "2024-04-30T11:22:27Z")

</div>

Hello,

Thanks for the quick response, here are the info you asked, don’t hesitate if you need anything else.

1. Get /\_cat/nodes?v (sorry I wasn't able to copy as a table nor as a correct image..). We can see that the ram.percent is super high but i'm not sure if it is really an issue after reading about that.

Heap.percent|ram.percent|cpu|Load\_1m|Load\_5m|Load\_15m|Node.role|Maste|  
49 100 0 1.23 0.91 0.86 rw -  
86 100 0 1.77 2.24 2.11 rw -  
34 85 1 1.69 1.40 1.43 cr -  
45 99 0 1.94 1.86 1.70 mv -  
20 99 14 4.79 4.20 3.74 himrst \*  
51 100 12 2.51 3.32 3.60 himrst -

1. GET /\_cat/health?v

| status | node.total | node.data | shards | pri | relo | init | unassign | pending\_tasks | max\_task\_wait\_time | active\_shards\_percent |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| green | 6 | 8 | 254 | 127 | 0 | 0 | 0 | 0 | - | 100% |

1. GET /\_cat/indices?v

I have 91 indices but here is a representative example (the issue is present for queries on both type of index, video metadata and motion events (data stream))

| health | status | index | pri | rep | docs.count | docs.deleted | store.size | pri.store.size / dataset.size |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Green | open | Index-type-1 | 1 | 1 | 193079552 | 0 | 60gb | 30gb |
| Green | open | Index-type-2 | 1 | 1 | 9313735 | 60344 | 2.5gb | 1.2gb |

1. General information

- Hardware profile: General purpose (was using Storage Optimized before and had the same issue)

- Global Memory (it’s a test system so we currently don’t have much warm/cold)  
 ![conf](https://us1.discourse-cdn.com/elastic/original/3X/0/5/05f1e3d60bb833abf3a7b37ba12d37c211d93af4.png)

- I just tested without the Size parameter and the results are probably a bit better but overall similar (it’s complicated to be sure as the query time is very unpredictable but I still had a index taking 7s to answer the time range query).

Thanks,  
Paul
