I'm having almost exactly the same issue. I can see how this happened but I can't see a way out of it.
In my case, it's because the original metricbeat indexed index (!!) had been removed due to space issues. So the various (several hundred) metricbeat agents come in and try and write to the alias. But there is no alias so the index is created with the name of the alias. This causes ILM to break.
I've tried to delete the index but of course it just comes back again because one of the several hundred agents just re-creates it.
Is there some way to "pause" ingestion on the server side? If so, I can then manually re-create the correct index name and alias and everything should be good. Stopping all of the clients is pretty well impossible across several hundred agents so really looking for something on the server side.
@dnorth98 , thanks for the tip - that got me moving in the right direction. If switching off action.auto_create_index in the metricbeat.yml instead of the user override, you must comment it out rather than setting to false, as this is too restrictive according to the error I received when I first set it false. I then deleted the metricbeat-n.n.n index, and the correct dated index with non-dated alias was created on the next agent connection after restart.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.