Response time spike at fixed interval of every ~10 minutes

Elastic Stack Elasticsearch
1

we are seeing one weird issue where the response time spike around every 10 minutes and same time the GC is getting triggered.

any help or information i can share would very helpful.

we are using version 8.12

Please let me know if i can check any configurations or share any data which help analysing this issue

2

Do you have monitoring enabled for your cluster? If so you can check the java heap utilizations and other memory utilizations and see if there are any correlations between high heap/memory usage and the GC.

For example, if heap is running out the GC process will be triggered and if there resourses are strained on your cluster, this might cause the cluster to be busy with GC instead of serving requests, thus increasing the response times.

3

we do have monitoring enabled and the heap usage is well under the max limit. we use Azul zing jvm in our cluster setup.

we do see below warning continuously.

[2024-09-10T11:26:04,878][WARN ][o.e.m.j.JvmGcMonitorService] [prod-test-instance] [gc][60948] overhead, spent [1s] collecting in the last [1s]

strange thing is it happens exactly at 10 minute intervals and i guess something is triggering this

Please do let me know if i can look into any other aspects.

4

Whats the status on CPU load and resources? 1s for GC sounds very long which makes me think it might be resource constrained.. In our environments usually long GC times are always connected to a high usage of CPU resources, either due to a core being saturated, or to the load being too high

5

According to the official support matrix it does not, if I am reading it correctly, seem like Azul Zing has been supported for quite a while. Is this issue reproducible with the bundled/supported JVM?

Do you have any external process pulling data every 10 minutes that might be resource intensive and trigger this?

6

@joean407 CPU was at ~44%. heap utilisation is at ~70%

@Christian_Dahlqvist we don't have any external process pulling data every 10 minutes.
yes i also gone through the support matrix and it says it doesn't support Azul Zing.
we haven't tried with any other JVM till now