ELK Stack - not creating indexes by day

Elastic Stack Elasticsearch
1

Hi!
I'm not sure if this should be here in this category or over in Logstash, but here goes. I've brought up a new 7.2 ELK stack cluster and have started ingesting data. When I started Logstash, it created the logstash-2019.07.15-00001 index and all was good. I was expecting to see a new index yesterday - logstash-2019.07.16-00001 but that didn't happen. Data was all still going into the 2019.07.15 index. After some searching I thought it might be the default ilm policy (logstash-policy) but I changed that to be 1 day, thinking that would rotate the indexes correctly but everything is still going into the 2019.07.15 index.
I did change the default index template to increase the mapping fields to 4000 so I'm not sure if that messed it up or not.
What can I do to get my indexes to rotate daily? I'll be glad to provide any logs, but so far none have really shown any errors that I can find. The cluster is working and happily ingesting log data, other than this one problem.

Thanks for the help!

2

Fixed this. I needed to disable ilm from logstash to make elasticsearch create the indexes by day properly. So my logstash configuration ended up like this in the output section

output {
if "_grokparsefailure" in [tags] {
stdout { codec => rubydebug }
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
ilm_enabled => "false"
}

3

If you want date math in ILM, you have to do this method when you create the initial ILM "bootstrap" index: https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html#date-math-index-names

The date will be when the new index is created, so today, 2019.07.25-000001, if it went a week to rollover, the next would be 2019.08.01-000002

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.