A few months back, I deployed the Elastic-Search(version - 8.0.1) on Kubernetes as a service as External load balancer using this guide.
Now, I am unable to perform any read or write operation on ES. I checked the logs, in which I found that memory of the node is almost full.
Here are some logs which support this analysis -
- flood stage disk watermark [95%] exceeded on [hulk-es-default-0][/usr/share/elasticsearch/data] free: 18.5mb[1.8%], all indices on this node will be marked read-only
- Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[1][0]]]).
- This node is unhealthy: health check failed on [/usr/share/elasticsearch/data].
Here are the errors that are coming when performing any read/write operation -
- elasticsearch.exceptions.TransportError: TransportError(503, 'master_not_discovered_exception', None)
- elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='...', port=****): Read timed out. (read timeout=30))
I don't think increasing the allocated storage space to the ES node is a long-term solution. Rather auto-scaling should be enabled.
Can anyone guide me how to resolve this issue and whether my approach towards solution is correct or not?