I have discovered that, every day at around 10AM (we are on UTC+2h, so UTC time would be 8AM) elastic cluster nodes iowait rapidly increases, including weekends, where there are not that much load at all. I do notice that behaviour also on our dev and test environment clusters at around 10AM.
Memory is not an issue, system is not swapping, I have checked graphs.
Elasticsearch runs on virtual machine (it is hosted by HPE Simplivity nodes, where all datastores are on SSD's).
Of course one might say that (since on HPE simplivity hosts, there are many VM's hosted), probably some other VM does something resource intensive at that time. However it might be, I have checked other VM's iowait as well at around 10 AM, and others seems fine at that time.
Our elasticsearch VM's are also on different HPE simplivty nodes and datastores, but each of them shows that something happens around 10AM each day. This can't be coincidence.
From kibana (20.03.2019 ~10AM)
Today (21.03.2019 ~10AM), same thing: