We see the same kind of problems for version 7.10. Either it does not roll over, or it does not list newly created indices as managed, or it somehow ignores the delete action completely if it changed the phase for one existing index. We tried to put the delete action back into the policy again, as suggested for an issue found on github, but that also does not work. ILM is in the RUNNING state. The Beats create the indices again, but somehow only sometimes create the index alias with a date. Nothing works as expected.
Thanks @antonkoenig. Does anyone else have the same problem or a solution to the problem/my config? This is a serious problems with large quantities of logs.
Hi @warkolm, I had a look at my setup but I can't see anything out of place. Do you see anything odd in the way I setup the rollover and the outputs above? I could do a manual rollover but I think that will just delay the problem.
Maybe the problem is the way we set it up? Right now it runs in Kubernetes and we have an initContainer creating the policy, index and alias. Once done, we spin the actual container that then finds the setup is present and directly starts writing to the index.
Here are some examples, first cleaning up, while metricbeat beats are already running and send in data, you might see the problems yourself when you try different combinations:
I did the test this morning. I applied your custom policy and custom template to a local stack and added an index as shown bellow
PUT metricbeat-000001/_doc/1
{
"title" : "How to Ingest Into Elasticsearch Service",
"date" : "2019-08-15T14:12:12",
"description" : "This is an overview article about the various ways to ingest into Elasticsearch Service"
}
I think the rollover time can be configured with a scheduler. It may have 10 minutes as default. So 1 minute may be too short, but this short time span is for testing purposes.
When we test these configurations, sometimes the index does not show that it's managed or other problems show up. Sometimes the index is in the delete phase, but it does not delete. Maybe some elastic guru knows what's going on?
I tried one of the guides from the elastic docs, but that guide creates a data stream. Our example does show the cleanup DELETE statements for that case also.
If ILM does not work, we may have to use Curator or something else. But as I understand it, this is a job for ILM and it should work already. Sadly it does not work.
The following worked for me. I set the max docs to 1000 and did have an index of 100,000 but at least it ended up doing a rollover of the index. The commands that helped me get the config working were GET gold-000011/_ilm/explain which gave me a status on the index state and POST /gold/_rollover which allowed me to see the error I was getting when trying to rollover.
My final (and pretty much uncahanged from my first post) is as follows. I hope it can help someone.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.