Implementing Rollover+ILM with Logstash and time based index name

Elastic Stack Elasticsearch
1

Hi there,

I used to index logs with Logstash into time-based indices (logstash-YYYY.MM.DD). Time-based index names make administration easier on us.
We already use lifecycle management to pull the indices to slower nodes and delete the indices after a certain amount of time. Now I want to take advantage of rollover to decrease index size.
I looked in the Rollover documentation and also have a basic understanding, but could not quite grasp how to implement this in my case.

I have an index template set up for logstash-*. Included is an ILM-policy.
When trying to set up rollover in the ILM-policy in Kibana I get a prompt asking me for an "alias for rollover index". I suppose this is the alias that gets applied to the current write-to index?:
grafik
I would now want to enter some sort of pattern like "logstash-YYYY.MM.DD" to state my time-based index pattern, because Logstash shall always write to "logstash-[current-date]" even if in the backend indices are named logstash-[current-date]-00004.

Am I still on the right path?

BTW: This is how my Elasticsearch output in Logstash looks like:

 elasticsearch {
            id => "12341234"
            hosts => ["192.168.1.2:9200"]
    	    user => "my_admin_user"
            password => "his_password"
            ssl => "true"
            ssl_certificate_verification => "false"
            document_type => "_doc"
    	    template_name => "logstash"
    	    index => "logstash-%{+YYYY.MM.dd}"
 }

Component versions:

  • Logstash: 6.8.8 (need to upgrade soon)
  • Elasticsearch: 7.9.0
  • Kibana: 7.9.0
2

The idea with ILM is that you write to the alias rather than a specific index pattern, and then behind the scenes it does the rotation. This removes the traditional approach of having a date in the index name entirely.

In your case you may want to setup the index pattern to be logstash-write, or whatever you want.

3

Hmm yeah, that would be the way to go then.
But in my opinion this makes management of the cluster not so straight forward, or did I miss something?
Here's an example:
In our case we usually close older indices to save resources.
Sometimes though we need to look at specific data again. With a day-based index name I know exactly which index to open.
How would I know which events withing a specific date range are in which index?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.