Rollover daily at night 12 am

Elastic Stack Logstash
1

I have created rollover indices for different applications, as I started the logstash service file at 8 PM, all the indices are getting created at that time only. However, there is a data mismatch, for instance: if the index is created on 24 March 8 PM, then in that index data of 25th March is also coming. This is the main reason I want them to rollover daily at midnight so that the data for a particular date is ingested in the correct index.

2

If you want to set the index name based on the timestamp of the event then why are you using rollover? What benefit do you hope to gain from it?

3

Can you provide more context about this? You created a rollover using ILM or just creating daily indices in the output?

Daily indices can be inneficient as they can be too small or too big, the recommendation is to rolover by size.

4

I'm not setting rollover based on the timestamp, it's based on date. The issue I'm facing is that I started the logstash service at 8 PM on 23rd March, so based on ILM rollover, it will generate a new index on 24th March at 8 PM. However, instead of generating at 8 PM, I want it to create at midnight so that the data of that particular date is written correctly in the index, for instance: uat-test-2025.03.23 should have only the 23rd March date data, not any other days' data

5

Perhaps I am misunderstanding your use case, but I would say you shouldn't use rollover at all.

6

Can you please share your Logstash output configuration and if you are using ILM your ILM configuration?

ILM is not based by date, it is based by age, if you configure it to rollover each 1 day, it will not rollover when the day changes, but when the age completes one day.

Also, even if you use daily indices, it will create a new index at midnight, but it is midnight at UTC (this cannot be changed), if you are on a different timezone it will rolover befor or after midnight on UTC.

7

I agree with you, @leandrojmp, that ILM is based on age. So, I think now ILM rollover for these indices is not necessary. However, can you let me know how to create indices daily at midnight?

input {
   http {
       port => 5044
       user => "elastic"
       password =>xxxxxxxx
       ssl => true
       ssl_certificate => "/etc/logstash/certs/logstash-dc.crt"
       ssl_key => "/etc/logstash/certs/logstash-dc.key"
       #codec => plain
       codec => "json"
       ecs_compatibility => "disabled"
    }
}

filter {
  mutate {
     rename => {"host" => "source_host"}
 }
}

output {
   elasticsearch {
     hosts => ["https://X.X.X.X:443"]
     user => "elastic"
     password =>xxxxxxxx
     ssl_certificate_verification => true
     ssl_enabled => true
     ssl_certificate_authorities => ["/etc/logstash/certs/elastic-dc.crt"]
     codec => "json"
   }
   if [logMessage][metaData][logSource] == "AppConnect" {
       elasticsearch {
               hosts => ["https://X.X.X.X:443"]
               index => "uat-appconnect-new"
               ilm_rollover_alias => "uat-appconnect-new"
               ilm_pattern => "{now/d}-00001"
               ilm_policy => "UAT_AppConnect_Policy"
               ssl_enabled => true
               ssl_certificate_authorities => ["/etc/logstash/certs/elastic-dc.crt"]
               user => "elastic"
               password =>xxxxxxxx
               }
       }
  else if [logMessage][metaData][logSource] == "OpenAPICGwy" {
       elasticsearch {
               hosts => ["https://X.X.X.X:443"]
               index => "uat-openapic-new"
               ilm_rollover_alias => "uat-openapic-new"
               ilm_pattern => "{now/d}-00001"
               ilm_policy => "UAT_AppConnect_Policy"
               ssl_enabled => true
               ssl_certificate_authorities => ["/etc/logstash/certs/elastic-dc.crt"]
               user => "elastic"
               password =>xxxxxxxx
               }
        }
   else if [logMessage][metaData][logSource] == "APICGwy" {
        elasticsearch {
                hosts => ["https://X.X.X.X:443"]
                index => "uat-enterprise-new"
                ilm_rollover_alias => "uat-enterprise-new"
                ilm_pattern => "{now/d}-00001"
                ilm_policy => "UAT_AppConnect_Policy"
                ssl_enabled => true
                ssl_certificate_authorities => ["/etc/logstash/certs/elastic-dc.crt"]
                user => "elastic"
                password =>xxxxxxxx
               }
        }
    else if [logMessage][metaData][logSource] == "ExternalAPICGwy" {
        elasticsearch {
                hosts => ["https://X.X.X.X:443"]
                index => "uat-external-new"
                ilm_rollover_alias => "uat-external-new"
                ilm_pattern => "{now/d}-00001"
                ilm_policy => "UAT_AppConnect_Policy"
                ssl_enabled => true
                ssl_certificate_authorities => ["/etc/logstash/certs/elastic-dc.crt"]
                user => "elastic"
                password =>xxxxxxxx
               }
        }
    else {
       elasticsearch {
                hosts => ["https://X.X.X.X:443"]
                index => "uat-other-new"
                ilm_rollover_alias => "uat-other-new"
                ilm_pattern => "{now/d}-00001"
                ilm_policy => "UAT_AppConnect_Policy"
                ssl_enabled => true
                ssl_certificate_authorities => ["/etc/logstash/certs/elastic-dc.crt"]
                user => "elastic"
                password =>xxxxxxxx
               }
   }
}

image
image435×508 19 KB