I'm experiencing very weird monitoring behaviour on two of our EK stacks. Yesterday on one server, and two days ago on another server, there was a 1,5 hour window of jitter visible on the graphs. If I zoom in to 10m, the lines are not connected. Some points are singular dots, some are L-signs.
One stack is r/w, another one is only being written to. The r/w has mappings, rollover alias and values are sorted, the w-only has nothing, just an auto-created index. The common thing between them is they're hosted on a single AWS instance each, with docker swarm consisting of 1 coord node, 3 masters, 3 data node, haproxy balancing 3 masters, and kibana accessing the coord node. There are plans to spread the nodes to different machines, but at the time it's not possible.
What is happening with these clusters and why?