Index roll over every 30 minutes

[zaeemmasood]

Hello All,

We have a logstash set up where input is based on a file.

Logstash parses the file and an index is created. A new file ends up for processing every 30 minutes.

We want a new index to be created every time a new file comes in (and the old one gets rolled over). This is needed to keep a historical record.

Please guide on how this can be achieved.

Thanks

[AquaX (Andreas Helmer)]

The index name needs to be somewhat unique in order for a new index to be created or your ilm policies need to be adjusted so that they rollover every 30 minutes.
The simplest way that I can think of is to put a timestamp in the index name in the logstash output.

 output {
    elasticsearch {
       index => "logs-%{+YYYY.MM.dd.HH.mm}"
    }
 }

A word of caution, as this can lead to a lot of shards being created. Do this at your own risk.

[warkolm (Mark Walkom)]

This isn't the best approach as you will end up with too many shards.

Why not just use ILM and then add the filename as a value inside the event?

[zaeemmasood]

Thanks @warkolm

I was able to make it work with what AquaX recommended above. Storage size of the index is roughly 450KB. I am using ILM to delete indices that are over 7 days old as that is what we require.

Can you please guide or provide a link on what you advise? especially the part requiring adding filename as a value inside an event.

Thanks

[warkolm (Mark Walkom)]

@zaeemmasood

Storage size of the index is roughly 450KB

That's a huge waste of resources.

[AquaX (Andreas Helmer)]

450KB??!
Yeah... that's incredibly small. You are much better putting everything into a single index and then creating ILM rules to manage what gets deleted and when. The sweet spot for the shards to be no more then 50GB (ideally between 10GB and 50GB as per

Size your shards | Elasticsearch Guide [7.15] | Elastic

This is probably one of the hardest parts in dealing with Elasticsearch (sharding and resource allocation) so there is lots of reading for you to do:

How many shards should I have in my Elasticsearch cluster?

[zaeemmasood]

Thanks for the detailed response @AquaX . I will definitely go through the links you posted.

The main reason of rolling over at a 30 minutes interval is to plot a graph and show the trend. Please see below:

Not sure if merging all in one index would allow me to do what I need?

I will also go through Tutorial: Automate rollover with ILM | Elasticsearch Guide [7.15] | Elastic to check.

Thanks

[warkolm (Mark Walkom)]

@zaeemmasood

Not sure if merging all in one index would allow me to do what I need?

That graph has nothing to do with index sharding, so yes, you can do that.

[AquaX (Andreas Helmer)]

You can do a date histogram for your data and set the bucket size to 30 minutes.

[zaeemmasood]

Thanks @AquaX and @warkolm

It seems that using Lens Line chart below only suffices my needs as it breaks down the display further with string serverName.keyword. Please see below for index created with hour and minutes (demo-csv-%{+YYYY-MM-dd_hh.mm}). This however creates too many shards:

I am not able to create the same if I take away hh.mm and use demo-csv-%{+YYYY.MM.dd} instead. This would of course create one index per day and the expectation is to plot data (same as shown in picture above) values based on 30 minutes time difference.

We are running ELK 7.6.2 stack.

Please guide

[zaeemmasood]

Hi All,

I was able to create a single index and it is now being rotated every 30 minutes through ILM. Indices are still being created every 30 minutes. Please see below:

Please let me know if this is the right way to move forward?

Thanks