Hello,
We are having issue with Hot tier in our elasticsearch version 7.16.2
Time to time in last few weeks we are experience issue that Metrics beat stop getting any data of the cluster.
What we are seeing in master is that some of nodes failed with:
Failed to retrieve shards from node.
"type": "server", "timestamp": "2024-06-18T14:27:34,223Z", "level": "WARN", "component": "o.e.c.InternalClusterInfoService", "cluster.name": "eck-elasticsearch", "node.name": "eck-elasticsearch-es-master-2", "message": "failed to retrieve stats for node [nodename]: [eck-elasticsearch-es-hot-v8-15][1.1.1.2:9300][cluster:monitor/nodes/stats[n]] request_id [272221826] timed out after [15007ms]", "cluster.uuid": "sdasdasd", "node.id": "sddsaddvdNOTA2Kin37RdKnxw" }
When we restart a pod issue get fixed temproary.
What we found is time to time our nodes hit max Peak Throughput 1200 MB/s combined for Read/Write
We are using pd-ssd tier on Google cloud n1 tier which one of the highest.
We also have been set on our indices index.refresh_interval to be 30 sec.