In Active Master node. we keep having the logs of below example. This happens to all indices and all files in /var/lib/elasticsearch/nodes/0/indices/...
- We are using local disk.
- Disk usage is not full.
- We tried to use the user "elasticsearch" and we are able to have write permission in /var/lib/elasticsearch
- We tried to switch to another Master node, these WARNING logs will then continue to show up in that active Master node.
- We tried to check the number of open files, it's lesser than 1000 so will be fine.
- We tried to reboot the Master node but it doesn't help.
1 * Load Balancing node
3 * Master nodes
3 * Data & Ingest nodes (Hot)
3 * Data & Ingest nodes (Warm)
The cluster is still running fine as of now, it's still able to do the indexing of new data and able to query old data as well. The only impact is that the status of all indices shown in Monitoring page, will show as "Unknown".
Does anyone of you encounter similar problem before? The problem started 2 days ago, although it causes not much of harm but the Elasticsearch logs keep growing (~100GB a day) and we are not able to check the indices status in Monitoring page.