We have set up our production environment with a 5 ES node cluster (3 hot and 2 warm) with a curator executing cronjob to push older logs into the warm nodes. As per industry regulations, curator password expired and was not reset on time, hence older logs filled up the data storage in the hot nodes. Even after password reset and moving older logs into warm nodes, logstash is unable to push the logs into elasticsearch.
Attaching the screenshot of the error at logstash and elasticsearch.
Please do not post screenshots. That screenshots is hard to read and does not contain the full exception as it is cut off.
Use markdown for proper formatting of snippets here.
My suspicion is, that you ran out of diskspace, where elasticsearch sets indices to readonly automatically. See https://www.elastic.co/guide/en/elasticsearch/reference/7.6/disk-allocator.html for more info, there is also a mention how to unset this setting (which happens by default on 7.6, but not on some older releases).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
