Daily Indexing (at Midnight)

Elastic Stack Elasticsearch
1

Hi there,

As I understand from this GitHub post and this Elastic Discuss post about having a new daily index (at midnight for my case), the solution to it is not Index Lifecycle Management. Instead, it should be using date math index.

I am trying this out, and using Winlogbeat as a Beats example.

I would like to ask if it is just the simple step of adding these 3 lines in my winlogbeat.yml configuration file?

Line #1:
setup.template.name: "winlogbeat-test"

Line #2:
setup.template.pattern: "winlogbeat-test-*"

Line #3:
output.elasticsearch.index: "winlogbeat-%{[agent.version]}-%{+yyyy.MM.dd}"

The documentation reference page which I looked at for this is found here.

(I have no need for a custom name, and I would like to have the agent's version as part of the index name.)

2

Is there a reason for not using ILM?

3

It was mentioned in the Elastic Discuss post that:

I agree with it (its reasoning appeared sound to me). Moreover, what ILM offers is an age-based rollover, which isn't what I am looking for.

4

Yep, but I am just looking for why you want date based indices?

5

Right. Thank you for the clarification - this might sound silly, but I would like to find out how many documents are held in an index each day and the size that a daily batch worth of document's holds.

6

That's not silly, makes total sense :slight_smile: It's a limitation of ILM.
FWIW I raised Show per day size stats for ILM indices · Issue #100009 · elastic/kibana · GitHub to see if we can do something here.

Basically if you want this via Beats, just disable the ILM portion of the config.

7

Just found out that interestingly someone else raised the same underlying question just very recently!

Thank you so much for this. :slight_smile: