Logstash index setting now requires version

[raamee]

Hi,

I am trying to upgrade my ELK setup from 5.6.5 to 6.2.1. So I am with the help of interactive guide Upgrade to Elastic 8.11.1 | Elastic Installation and Upgrade Guide [8.11] | Elastic, checking the manual upgrade steps.

I have a doubt in Filebeat breaking changes:

If you use the Logstash output to send data from Beats to Logstash, you need to update the index setting in your Logstash configuration to include the Beat version:

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Prior to 6.0, the recommended setting was:

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

The index templates that ship with 6.0 are applied to new indexes that match the pattern [beat]-[version]-*. You must update your Logstash config, or the templates will not be applied.

I have the following config in my server:

output {
if [type] == "apache-access" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => false
manage_template => false
index => "apache-access-%{+YYYY.MM.dd}"
}
}
else
{
elasticsearch {
hosts => ["localhost:9200"]
sniffing => false
manage_template => false
index => "apache-error-%{+YYYY.MM.dd}"
}
}
}

So what should I change? Should I mention the Filebeat version number? ( just 6.2.1 )? so that it becomes:

index => "apache-access-6.2.1-%{+YYYY.MM.dd}"
index => "apache-error-6.2.1-%{+YYYY.MM.dd}"

Is this fine? or Am I wrong? Won't this break existing indices?

Thank you

[raamee]

Hello,

Any one? I'm stuck here and due to these, I am unable to upgrade to ELK 6.2.2 completely. I only upgraded Elastic and Kibana.

Thank you.

[ruflin]

You can use the variable recommended in the migration guide to not have to hardcode it.

But looking at your config with the apache prefix it looks like you already use a special index pattern which will not match the index template from filebeat anyways. How do you load the index template?

[raamee]

Hi Ruflin,

First of all, thank you for replying.

Yes, this is a custom log and I am not using the template from filebeat. I send the logs from filebeat to logstash where the fields are getting loaded and then fed to Elasticsearch.

Thanks again.

[ruflin]

If you have your own template and index names anyways you don't need the beat version. In this case we assume you know pretty well what you are doing