Daily rollover with ILM

Elastic Stack Elasticsearch
1

Hi,

I have an index named "winlogbeat-7.1.1-2019.07.02-000001" and I want to rollover every day. I have define a lifecycle policy for rolling when the index is bigger than 10GB, but I don't know how define the daily rollover. Tomorrow should create a new index called "winlogbeat-7.1.1-2019.07.03-000001". ¿Any idea?

Thanks in advance!

2

Hi Juan,

In order to roll over daily, you can add the "max_age" parameter in your ILM configuration, see: https://www.elastic.co/guide/en/elasticsearch/reference/7.2/_actions.html#ilm-rollover-action for the supported parameters.

May I ask why you want daily indices though? It's much better to roll over on conditions like maximum number of documents or shard size, since then you don't create a bunch of smaller indices on days that don't receive a lot of new documents.

3

Hi Lee,

Thanks for the response. If I use "max_age" parameter setting it to 1 day, the roll over is performed when the index is 1 day old, but if the index is not created at 00:00, the solution is not adequate.

Every day I received about 5GB in an index, and I think it can be a good option to roll over daily.

4

Can you explain why this does not work for you?

Every day I received about 5GB in an index, and I think it can be a good option to roll over daily.

It would be better to roll over based on size rather than daily, 5gb is a rather small index, we usually recommend something like 20-40gb (depends on the number of primary shards). By rolling over based on size you can reduce the number of indices to only what you need, which will be beneficial for the cluster long-term.

5

I have not experience with sizing indexes and surely you are right when yo say that is better to roll over based on size.

Otherwise, I will explain why is not working for me the use of "max_age" parameter setting it to 1 day. If I create and index named 'my_index-2019.07.31' at 11:07AM the next day index named ''my_index-2019.08.01'' will be created at 11:07AM of 2019-08-01, so the logs from 00:00AM to 11:07AM of 2019-08-01 will be stored in the previous day index.

But with your explanations I understand better how to use the ILM.

Thanks @dakrone !!!

6

I think it rolls at at the 1 day interval from index creation, so if the original index wasn't created at 00:00, say at 00:16, the day + 1 will be created when the first event after 00:16 is indexed. If an index isn't receiving events, it won't roll.

If you want 00:00 daily, don't use ILM, go back to having the yyyy.dd.mm in the index name. Note that if you set @timestamp to field data, if you get old data, it will go into old indexes. Say you enable unrestricted winlogbeat on a 4 year old windows box, you will likely create 1400 date-stamped indices just getting caught up to current data.

7

Just for my understanding, is it expected (by Kibana or elasticsearch) that an index named with yyyy.mm.dd contains documents only of that date?

8

No, Kibana doesn't use the index name for any sort of calculations, so it isn't expected that the dates of the data match the index name.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.