Illegal_argument_exception: index.lifecycle.rollover_alias does not point to index

Elastic Stack Elasticsearch
1

Hi there,

I am getting ILM errors on all of my indexes. For example:

illegal_argument_exception: index.lifecycle.rollover_alias [ucv-voice-events-cube] does not point to index [ucv-voice-events-cube-2020.08.28]

In my case both the index and the alias exist. The ILM profile should roll over each index at max 50GB or 30 days. However the indexes are only rolling over each day.

I think that the problem is I'm not appending -000N to the end of the index. Does anyone know the custom Logstash pattern to get this working? Right now we are consuming from a kafka topic and using that name and the data to create the index. It does roll over every day, but I cannot bind it to an ILM profile to keep the index open until 50GB size is reached and then delete it after 15 days have passed.

output {
    elasticsearch{
    	hosts => "${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"
    	index => "%{[@metadata][kafka][topic]}-%{+YYYY.MM.dd}"
    }
}
2

Also a previous question: Does ILM work in Logstash with custom index names?

And the issue I raised as the ilm_pattern => "000001" didn't work either:

output {
    elasticsearch{
    	hosts => "${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"
    	index => "%{[@metadata][kafka][topic]}-%{+YYYY.MM.dd}"
#    	ilm_rollover_alias => "%{[@metadata][kafka][topic]}"
#    	ilm_policy => "%{[@metadata][kafka][topic]}"
    	ilm_pattern => "000001"
    }
}

I opened an issue here: https://github.com/elastic/logstash/issues/12214

I saw that @warkolm was addressing a similar question here: How to correct custom logstash index name for use by ILM

2 Likes
3

I am running into a very related issues. I hope there is some clarification on this soon.

1 Like
4

Check out https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm for how Logstash works with ILM.

5

I did read that documentation, but unfortunately the example there didn't work (see above) and there is no good, working example on how to do Logstash + custom index + ILM. If I uncomment the lines in my example above (to make custom aliases and policies), I get an exception. As-is Logstash ignores the ilm_pattern => "000001" and I just get the default YYYY-mm-dd daily rollover However when it tries the ILM profile I get the error: " Illegal_argument_exception: index.lifecycle.rollover_alias does not point to index" probably because there isn't the expected -00000N after the index name-date format.

Default filebeat, metricbeat and Logstash ILM finally works in current ElasticStack versions (this wasn't the case for several releases) but I cannot figure out how to get custom Logstash indexes to work with ILM despite reading all the documentation, tutorials, and stack overflow and this forum.

I think I've got to believe that it doesn't work and I'll just have to manage the life of my daily indexes with an external cronjob and the API.

Thanks for the suggestions though, I just wish I could get this working as-built in ElasticSearch.

6

It looks like the section you are missing is https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-alias-bootstrap

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.