This morning I woke up to this:
This is not the first time that it happens, and when it happens it blocks everything: I cannot access Kibana, Elasticsearch doesn't answer at all, this means I also cannot delete data using the API nor scale the instance to give it more disk space.
My only fix is to usually destroy the whole deployment and start over ...
My current set-up is the following:
I have 2 logstash sending logs from 2 kubernetes cluster (using filebeat). The logstash output is the following:
output {
elasticsearch {
hosts => ["<elastic search url>"]
user => '<user>'
password => '<password>'
data_stream_namespace => 'production'
}
}
They both write to a data-stream. In kibana I see that this data stream is linked to the ILM for logs. By default, this ILM keep everything forever. I change this policy to move the data to a cold storage instance after 2 days and delete it after seven days.
My issue is that this ILM doesn't seem to work / be respected. This is my cold storage instance after more than 2 days:
No data seem to be written to it. In the same way, I used a ILM before were data should be deleted after 2 days and after 2 days, no data seem to have been deleted in the same way.
Therefor I have 3 questions:
- Is there a way to debug my instance when it reach 100% disk ? I am using ElasticCloud
- Is there a configuration to make the instance read-only when it reach 95% ?
- I am missing something with data-stream and ILM ? Why the ILM doesn't seem to work ?
Thank you