Index roll over every 30 minutes

Elastic Stack Logstash
1

Hello All,

We have a logstash set up where input is based on a file.

Logstash parses the file and an index is created. A new file ends up for processing every 30 minutes.

We want a new index to be created every time a new file comes in (and the old one gets rolled over). This is needed to keep a historical record.

Please guide on how this can be achieved.

Thanks

2

The index name needs to be somewhat unique in order for a new index to be created or your ilm policies need to be adjusted so that they rollover every 30 minutes.
The simplest way that I can think of is to put a timestamp in the index name in the logstash output.

 output {
    elasticsearch {
       index => "logs-%{+YYYY.MM.dd.HH.mm}"
    }
 }

A word of caution, as this can lead to a lot of shards being created. Do this at your own risk.

3

This isn't the best approach as you will end up with too many shards.

Why not just use ILM and then add the filename as a value inside the event?

4

Thanks @warkolm

I was able to make it work with what AquaX recommended above. Storage size of the index is roughly 450KB. I am using ILM to delete indices that are over 7 days old as that is what we require.

Can you please guide or provide a link on what you advise? especially the part requiring adding filename as a value inside an event.

Thanks

5

That's a huge waste of resources.

6

450KB??!
Yeah... that's incredibly small. You are much better putting everything into a single index and then creating ILM rules to manage what gets deleted and when. The sweet spot for the shards to be no more then 50GB (ideally between 10GB and 50GB as per

This is probably one of the hardest parts in dealing with Elasticsearch (sharding and resource allocation) so there is lots of reading for you to do:

7

Thanks for the detailed response @AquaX . I will definitely go through the links you posted.

The main reason of rolling over at a 30 minutes interval is to plot a graph and show the trend. Please see below:

image
image982×583 59.8 KB

Not sure if merging all in one index would allow me to do what I need?

I will also go through Tutorial: Automate rollover with ILM | Elasticsearch Guide [7.15] | Elastic to check.

Thanks

8

That graph has nothing to do with index sharding, so yes, you can do that.

9

You can do a date histogram for your data and set the bucket size to 30 minutes.

image
image1907×727 65.9 KB

10

Thanks @AquaX and @warkolm

It seems that using Lens Line chart below only suffices my needs as it breaks down the display further with string serverName.keyword. Please see below for index created with hour and minutes (demo-csv-%{+YYYY-MM-dd_hh.mm}). This however creates too many shards:

tempsnipZZ
tempsnipZZ1527×613 98.1 KB

I am not able to create the same if I take away hh.mm and use demo-csv-%{+YYYY.MM.dd} instead. This would of course create one index per day and the expectation is to plot data (same as shown in picture above) values based on 30 minutes time difference.

We are running ELK 7.6.2 stack.

Please guide

11

Hi All,

I was able to create a single index and it is now being rotated every 30 minutes through ILM. Indices are still being created every 30 minutes. Please see below:

image
image1197×866 75 KB

Please let me know if this is the right way to move forward?

Thanks

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.