[include_in_all] is not allowed for indices (after upgrading to 6.0.0)

[dulman]

Hello,

we have upgraded our Elastic Stack (Elasticsearch, Logstash, Kibana, Beats) to version 6.0.0.

Our Logstash config:

input {
tcp {
   type => app
   port => 5140
 }
}

output {
	elasticsearch {
            hosts=> ["elasticsearch-endpoint"]
            index=> "logstash-dev-elastic01-app-%{+YYYY.MM.dd}"
    }
} 

With this config we are running into mapper_parsing_exceptions:

[2017-11-30T14:06:14,283][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-dev-elastic01-app-2017.11.30", :_type=>"app", :_routing=>nil}, #<LogStash::Event:0x16db134e @data={"@timestamp"=>2017-11-30T14:06:14.267Z, "port"=>23639, "@version"=>"1", "host"=>"ip-00-00-00-00.eu-central-1.compute.internal", "@metdata"=>{"ip_address"=>"00.00.00.00"}, "message"=>"{\"hostname\":\"app1\",\"env\":\"him1\",\"level\":\"info\",\"message\":\"executing job close-incidents-job\",\"label\":\"cron\"}", "type"=>"app"}>], :response=>{"index"=>{"_index"=>"logstash-dev-elastic01-app-2017.11.30", "_type"=>"app", "_id"=>nil, "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [_default_]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."}}}}}

How to solve this problem?

[magnusbaeck (Magnus Bäck)]

It seems like the index template being applied is too old for your ES instance. What does your index template look like?

[dulman]

Ok, I had to delete the logstash template manually.

curl -XDELETE 'http://localhost:9200/_template/logstash'

Thank you!