Exiting: resource 'metricbeat-7.8.1' exists, but it is not an alias

[twilight (Aatif)]

Hello,
Our ES cloud instance was having some storage issues, in order to clear some space we removed the metricbeat data.

Now some of our metricbeat agents are not able to log data into metricbeat-* index, I tried to run the 'metricbeat setup' but that failed with a message: resource 'metricbeat-7.8.1' exists, but it is not an alias.

Under index management, I can see an index named 'metricbeat-7.8.1'. I noticed it does not have any lifecycle policy attached and when I try to attach, it says there must be some alias for this index.

It means that the index I am seeing has a wrong name, there must be some alias of the same name and the index should have name with date and time like metricbeat-7.8.1-2022.02.03...

How can I fix it?
Thanks

[twilight (Aatif)]

This is how we fixed it;

Stop all metricbeat agents which are logging data to metricbeat-* index. Reason is they create the metricbeat index again but with a wrong name.

Once the agents are stopped, delete the index metricbeat-7.8.1 (it will lose all metricbeat data) and make sure it is not created again instantly by some agent.

Now the metricbeat index is deleted, run metricbeat setup in shell, that would create the index and aliases properly. This time it would run successfully since the index does not exist so it will proceed creating one.
When it is done, you can check in Kibana the new index with alias and a lifecycle policy is created. Great!
(some people suggested to create the index and alias manually, I think the setup is a safer and proper way to do the same)

Now start the metricbeat agents again, you will start seeing data from all of them.

Note: to make sure all agents would work fine with newly created index, you can run the 'metricbeat setup' on each node / machine where metricbeat is running, before starting the metricbeat service.