Elasticsearch nodes iowait every day at around 10AM (UTC time 8AM)

Elastic Stack Elasticsearch
1

Hi

Elastic 6.6.1
Centos 7.6

I have discovered that, every day at around 10AM (we are on UTC+2h, so UTC time would be 8AM) elastic cluster nodes iowait rapidly increases, including weekends, where there are not that much load at all. I do notice that behaviour also on our dev and test environment clusters at around 10AM.

Memory is not an issue, system is not swapping, I have checked graphs.

Elasticsearch runs on virtual machine (it is hosted by HPE Simplivity nodes, where all datastores are on SSD's).
Of course one might say that (since on HPE simplivity hosts, there are many VM's hosted), probably some other VM does something resource intensive at that time. However it might be, I have checked other VM's iowait as well at around 10 AM, and others seems fine at that time.
Our elasticsearch VM's are also on different HPE simplivty nodes and datastores, but each of them shows that something happens around 10AM each day. This can't be coincidence.

Zabbix graphs:

image
image.png1507×717 43.9 KB

From kibana (20.03.2019 ~10AM)

image
image.png1661×832 124 KB

Today (21.03.2019 ~10AM), same thing:

image
image.png813×313 12.8 KB

image
image.png1657×840 120 KB

Regards
Raul

2

There is an increase and subsequent decrease in segment count starting at that time. Do you have any Curator jobs that perform forcemerge or shrink operations around that time?

3

I have explicitly created no jobs whatsoever.
So everything that is running on background, all is from elasticsearch's side (sth that comes with elasticsearch installation by default).

Raul

4

I am not aware of anything kicking off at any specific time that comes with Elasticsearch as default.

5

Hmm, ok, well then it is strange.
I actually did hope that there were something that is kicking off from elasticsearch's side by default at that specific time.

But then it is a mystery to me..

You referred to forcemerge or shrink. Couldn't that be that these operations are too drastic by default (I must say that I have not changed defaults in any way).

Regards
Raul

6

They are not triggered by default, which is why I asked about Curator which typically is run through a cron job.

7

We haven't created no jobs explicitly.

But I'll rephrase.

This merge operation (not forcemerge) shouldn't be something that runs on background by default..?
For example, when segment has more than X percent of deleted documents, then it will merge some segments automatically (as I have understood).

But however, still, every day around 10AM, different clusters. This just is odd ..:smiley:

Regards
Raul

8

Merging takes place in the background, but not at specific times.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.