Default Index Pattern (logstash-yyyy.MM.dd) Failing

[scriner]

Hi all. We are using the following pipeline config file to ship syslogs to Logstash:

input {
  tcp {
    port => 1514
    codec => "json"
    type => "syslog"
  }
  udp {
    port => 1514
    codec => "json"
    type => "syslog"
  }
}

filter {
  # This replaces the host field (UDP source) with the host that generated the message (sysloghost)
  if [sysloghost] {
    mutate {
      replace => [ "host", "%{sysloghost}" ]
      remove_field => "sysloghost" # prune the field after successfully replacing "host"
    }
  }
}

output {
  elasticsearch {
    hosts => ["{IP}:9200","{IP}:9200","{IP}:9200","{IP}:9200"]
    index => "logstash-%{+yyyy.MM.dd}"
  }
}

This works, but the index is not getting tagged correctly when we look at the documents in Kibana. If I change the index line in the output block to something else - syslog-yyyy.MM.dd, logstash-yyyy.MM.dda, etc. - then everything is fine and the indices are created with the current date as you'd expect. If I leave them as shown with the standard logstash-yyyy.MM.dd pattern, it is not respected. Right now it's just using an old date with -00001 appended to the end, which I'm guessing is a rollover index or something.

I guess we can just use a different index pattern like syslog like I mentioned above, but I really would prefer to know how to fix this.

Thanks!

[Badger]

@scriner

index => "logstash-%{+yyyy.MM.dd}"

If elasticsearch supports it (i.e. a recent version) then logstash will be using ILM and the index option is ignored.

[scriner]

I appreciate the reply - I think I've wandered down this path in the past but I must not understand completely.

When changing the Index Lifecycle Policy for the logs Managed Index Template (which is the closest thing I can find that might apply to my current predicament) from 30 days or 50GB to 1 day, the index remains logstash-2022.03.29-000001. And, if I understand right, this policy would have already rolled this index over a long time ago based on the 30 day policy. I don't understand how we've been using the same index for 6 months.

Ideally, today for example, each document sent would get tagged with index logstash-2022.09.26, and increment with the days. Is this not possible using the default logstash-yyyy.MM.dd format?

[stephenb (Stephen Brown)]

What version of logstash and Elasticsearch are you using?

And if I read it correctly you want daily indices? Is that correct?

And are you planning to delete them after a certain length of time?

[scriner]

We're currently at version 7.17.6.

And yes, that was what I wanted originally, but I think I've decided to just let ILM manage all of these for me and use the rollover pattern naming. I think the only reason we wanted daily indices is to be able to reliably expire them after they're a year old, but my understanding is that ILM can handle that using document data so I think doing it this way is fine.

[leandrojmp (Leandro Pereira)]

@scriner

but my understanding is that ILM can handle that using document data so I think doing it this way is fine.

The ILM will use the date in the index name as the creation date, for example for the index logstash-2022.03.29-000001 it will use 2022.03.29 as the date to calculate when it will move between the phases in the ILM policy.

[scriner]

I'm seeing our index come in as this: logstash-ilm-000001

Using this output block:

output {
  elasticsearch {
    hosts => ["{CLUSTER}"]
    ilm_enabled => true
    ilm_rollover_alias => "logstash-ilm"
    ilm_policy => "AnnualRotateRolling"
  }
}

Trying to set ilm_pattern as {now/d}-000001 as shown in the docs stops our logs from coming in.

[scriner]

This is now working as expected - removing the ilm_rollover_alias line allowed the index to include the date as expected. Thanks all!