Help With ILM

[Bruceclegg]

I still consider myself very much a neophyte with ELK.

A few weeks ago I set up Lifecycle Policies for Indices in Kibana. I used the old index names the developers have hard coded into their programs.

But when it came time for the rollover I checked and found the rollover had failed in the check-rollover-ready step. I got the error message: illegal_argument_exception: index name [aapc.log-test] does not match pattern '^.*-\d+$'

I did some googling and learned that I should have named the indexes with a number at the end. So instead of aapc.log-test, I should have named it aapc.log-test-000001 - and set the alias to be aapc.log-test.

My question is: What is the best way to get there from here?

I understand you can't rename indexes. The easiest solution seems to be to delete the indexes and build new ones with new names and better aliases. But I'd rather not delete if I don't have to. Is there a better answer?

Thanks

[Iker (Iker Saint)]

There are several options, you could do the following:

• Delete the index and build new ones with the correct name format.
• Use the reindex api to a new index with the proper name
• Use the clone api to "rename" the index (its a very fast process)
• Take a snapshot and restore it with the proper name, however there are a couple of options to check to really rename the index and avoid future issues
• Delete or reindex the current index infavor of DataStream that handles all the naming for you. Considering that you are ingesting logs, this could be your best option.

[warkolm (Mark Walkom)]

Can you share your policy?

[Bruceclegg]

Sorry for delay - was out the last few days.

{
"Index_lifecycle" : {
"version" : 1,
"modified_date" : "2020-09-02T18:42:20.699Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "30d"
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : {
"min_age" : "60d",
"actions" : {
"delete" : { }
}
}
}
}
}
}

[Bruceclegg]

There is much I don't know about this yet.

First I tried re-index

 I created an empty index
        PUT /aapc.log-test-00001

Then I attempted the reindex
POST _reindex
{
"source": {
"index": "aapc.log-test"
},
"dest": {
"index": "aapc.log-test-00001"
}
}

and got mapper_parsing_exceptions

I read on this a bit and it seemed cloning might be the better solution as it would build the new index with the same properties as the source.

But - reading about cloning - it tells me the index must be 'green' - mine are yellow because of the illegal_argument_exception with the index name issue with ILM. Is this really going to be a problem for cloning? It also says I've got to mark the index as read-only. I don't want to have to do that either, unless I don't have to.

I'm going to go back to seeing if I can create the new index with the same mappings as the source. I will post here with my results.