How can i modify the index name when using ILM?

Elastic Stack Elasticsearch
1

Hi,
i'm using ILM. I'd like to have index names which reflect the source of the data, e.g. index-mysql-hostname. I read that i can configure the index name in filebeat, but only without ILM.
Is there a way to create an index template with an index pattern and choose in filebeat (or even better in a filebeat module) this index ?

Thanks.

Bernd

2

Be aware that using this type of granular index naming convention can lead to a lot of very small indices, which is very inefficient and can cause performance problems. I would recommend grouping data into indices based on data type and retention period rather than source.

If you still want to do this it is common to dynamically create index names based on fields in the data. You can do this with ILM, but will not be able to use rollover as this type of indices as far as I know can not be dynamically initiated.

3

Hi Christian,
how can i do this ?

Bernd

4

You can configure index names based on fields in the data as shown in the Filebeat docs. Note there likely will be issues if the field(s) used does not exist for all documents. You can create an ILM policy and attach this to these indices through an index template, but this can not use rollover.

5

Well, ILM without rollover is senseless.

Can you point me to the documentation where i can find out how to do this ?

Thanks.

Bernd

6

No, it is not. It does assume you are creating time-based indices though, which can be done without rollover. ILM still allows you to manage the index lifecycle and transition indices between phases before deleting them.

I do not think there is any documentation about this. You need to look at the different types of data you have, determine how long you need to keep it and then map it to a suitable index yourself. How you do this will depend on your data and indexing pipeline.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.