[SOLVED] Index issues after upgrade from 5.x to 6.2

[Diggy]

Hello, all.

Yesterday, I upgraded my Elastic stack from version 5.x to 6.2, and today, my logstash and filebeat indices no longer work. Here's some output from elasticsearch.log:

[2018-02-08T00:03:29,150][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2018.02.08", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x6297d867], :response=>{"index"=>{"_index"=>"logstash-2018.02.08", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [default]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."}}}}}

I've googled extensively, but can't seem to come up with a solution. I would appreciate your (very detailed) help in getting back to good.

Many thanks.

[Enis]

Hey,

I had the same problem and the solution for this problem is written in this post:

Failed to parse mapping for index

Get the current one using the index template API and update it. You can the push it back into the cluster using the same API. If you are using the default Logstash template, you can just upload the standard 6.0 template and overwrite the existing one.

I have deleted my saved logstash template and added the standard logstash template like mentioned.

[Diggy]

Enis,

Thanks so much for the reply. I'm happy to say that your suggestion worked for logstash! Very much appreciated!

I still have the same problem for Filebeat, though. Any idea about how I deal with that?

Diggy

[Enis]

Sorry. I had that problem only with Logstash. For Filebeat it could be perhaps a separate template if you use Filebeat directly to write to Elasticsearch. Which version of Filebeat are you using and did you update your config file?

[Diggy]

Well, I think I've made some progress re: Filebeat, but I don't know what it is that I did. Filebeat log for one of many servers is now appearing in Kibana, but it's only the apache-access log. /var/log and /var/secure log information are not appearing. Nor are logs from any other server running Filebeat. Can anyone help?

[Diggy]

Well, solved mostly. I followed Enis' suggestion for the Logstash part, and that worked. For Filebeat, I had to upgrade to version 6.2 on one host, generate filebeat.template.json on that host and get it over to my Elastic host, remove the filebeat index, and add the index with the filebeat.template.json. I'm still struggling to get Filebeat working on my Windows hosts, but will ask about that in the Filebeat list.