Filebeat create new index at every x hours

Elastic Stack Beats
1

Hi all!

The storage under ELK is limited, and can't be increased. So I need to delete indexes very often but deleting one-day indexes would result in the unnecessary loss of a massive amount of documents therefore I need to slice one-day indexes into smaller indexes.

This is a part of my current Filebeat setup:

output.elasticsearch:
  index: "scapp-%{[agent.version]}-%{+yyyy.MM.dd.HH}"

so Filebeat should create new index at every new UTC hour and it works perfectly

I have ~120GB index size per day so one hour index size is far from optimal for Elasticsearch

I'm thinking about creating indexes every 3 or 4 hours.

Using Rollover API sounds reasonable BUT does not apply here, because it counts the hours relative from the beginning of the index creation and is not based on the UTC clock.

I read something about "date math" but I can't find docs about that (how) could I use it in the output.elasticsearch.index in the Filebeat config

Is it possible at all to setup Filebeat to create new index only every 2,3,4,x hours?

(Elasticsearch + Filebeat + Kibana all at latest version)

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.