Question about ILM

[borna_talebi]

Hi,
I'm using a custom index in my filebeat so I had to disable ilm in elasticsearch.
My question is if there is a automatic solution to delete indices older than 3 months when ilm is diabled? or maybe have a hot-cold architecture?
My filebeat indices are created monthly.

[Emanuil (Emanuil Tolev)]

Filebeat sets up ILM by default since 7.0 (as per docs).

I think what you mean by

I had to disable ilm in elasticsearch.

is that you're not using the default ILM config from Filebeat and have set setup.ilm.enabled: false in your filebeat config. But ILM is not actually a Filebeat feature, it's an Elasticsearch general feature. You can configure ILM for any index or group of indices in Elasticsearch, including your custom index. Look at the ILM feature docs rather than the Filebeat docs. You can set up a policy to delete indices after 90 days with ILM and apply it to your custom index.

If you share your filebeat config incl the custom index and the index name pattern it might be possible to help you more concretely.

[borna_talebi]

Hi Emanuil, thanks for your answer.

Here is my filebeat config.

@Emanuil

But ILM is not actually a Filebeat feature, it's an Elasticsearch general feature. You can configure ILM for any index or group of indices in Elasticsearch, including your custom index.

according to documents :

Custom index settings are ignored when ILM is enabled.

so I stopped the ilm service using POST _ilm/stop because before this command I couldn't see my custom index (cisco beats) in the index management.

[Emanuil (Emanuil Tolev)]

@borna_talebi

Custom index settings are ignored when ILM is enabled.

Yes, Filebeat ignores custom index settings when you turn on its ILM policy. But the ILM feature is much more general than how Filebeat uses it. That's why I directed you to the ILM feature docs as opposed to the Filebeat-with-ILM docs.

To apply an ILM policy to a series of indices, you ideally use a policy template. The Kibana screenshot of creating such a template manually shows you it has an Index Settings step where you can supply your custom settings within an ILM policy. Just below the screenshot the doc links to the PUT policy template API doc. The template object contains a settings object which allows any custom index setting.

settings (Optional, index setting object) Configuration options for the index. See Index Settings.

So, try setting up an ILM policy yourself and applying it to your custom index. Set setup.ilm.enabled: false in your Filebeat config to prevent it from using its own ILM policies.

EDIT: Between Filebeat's use of ILM and the general ILM feature this is a bit complicated. If you post you custom index settings here I'll show you how to define an ILM policy to "delete data after 3 months" as per your original post, and you can then customise it further when you confirm it works.

[borna_talebi]

@Emanuil

Yes, Filebeat ignores custom index settings when you turn on its ILM policy. But the ILM feature is much more general than how Filebeat uses it.

So if I just set setup.ilm.enabled: false i can use general ILM feature for other indices including my custom filebeat index, correct?

@Emanuil

f you post you custom index settings here I'll show you how to define an ILM policy to "delete data after 3 months" as per your original post,

Thanks but I think by following the docs I can configure it myself.

[Emanuil (Emanuil Tolev)]

@borna_talebi

So if I just set setup.ilm.enabled: false i can use general ILM feature for other indices including my custom filebeat index, correct?

You can always use the general ILM feature on other indices as long as you don't disable it via other means like _ilm/stop - even while Filebeat is using ILM. setup.ilm.enabled: false tells Filebeat not to use ILM which you need for your specific use case. By applying it, you can start ILM again, so you can then use it for a custom policy.

[borna_talebi]

Thanks a lot for your detailed responses.

Best regards,
borna