How to correct custom logstash index name for use by ILM

Elastic Stack Logstash
1

I have a custom logstash index named asa-%date%. I have attempted to apply an ILM policy to it, but I get an error when it attempts to apply. According to an elastic team member in another thread this is because ILM expects the index name to have a versioning number appended to the end. Such as asa-2020.08.25-00001

How do I apply the versioning "field" option to my custom name? The index for beats out of the box seems to apply this but the documentation simply states:

Optional index name. The default is "filebeat" plus date

and generates [filebeat-]YYYY.MM.DD keys.

Where and how does the versioning number get applied to the index name?

2

Sharing the error, and policy and template would be helpful.

3

Sorry, here is the other thread that details my problem exactly, with a hint at a solution but I unfortunately needed more details concerning fixing the index name:

Here is my error:

illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-7.8.0] does not point to index [asa-2020.08.18]

My simple policy is this:

PUT _ilm/policy/Delete-After-7 { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": {} }, "delete": { "min_age": "7d", "actions": { "delete": {} } } } } }

Not sure how to format about code, the back tic is not working for me. Apologies.

Also not sure how to post template as there is tons of information, however it is the default Filbeat 7.8 template with the only change that I applied the template to also include "asa-*" which is not functioning as far as ILM

4

Did you bootstrap the initial index?

5

I did not manually bootstrap the original index. The original index was simply a default setup of an Filebeat shipper with ILM enabled as part of the config so I do not know if bootstrapping automatically takes place as part of that process. The ILM process for that index appears to be working.

I assigned my non standard index to this default Filebeat 7.8 Index Template that was automatically created. I was hoping this would let it sort of piggy back on to the working index. Obviously it is more complicated than that.

6

Ok, you should be able to run something like;

PUT /my_logs_index-000001/_alias/logs

To assign the logs alias to the my_logs_index-000001 index. Just replace that with your index and alias as needed.

7

Sorry for the long delay but hopefully I can continue to work on this.

In regards to your latest response. My index does not increment like the example you have given. There is no -00001 appended to the end of the actual index name. In fact this index was originally created back in 5.x version of the stack and has simply been carried over through the different iterations.

I have a predefined ILM policy that will simply purge any documents after 14 days. It is tied to a default "filebeat-*" index via the filebeat template.

So I am not using a data stream, but an old school index. It already is using a pre existing filebeat index template, so I am not clear how that relates to this new index alias concept that I am not familiar with even after reading the documentation. Also, my index does not auto increment with a value on the end which it appears ILM requires.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.