Could not index event to Elasticsearch after upgrade 5.6.14 -> 6.6

[Slartibartfast (Dave Lewis)]

I have a very small Elastic Stack running one node and with LS and FB on the same server as ES. Seven other servers kick syslogs and audit logs to LS via FB. When I inherited the setup, it was running ELK 2.x and I successfully upgraded to the 5.6 version and let it stew on that version for while before heading up to 6.x.

Last week, I got everything upgraded and the logs were streaming in just fine until ~2000 at which point the whole thing stopped. I found that I had some unassigned shards and I assumed that the problem might be there so I removed those indexes in hopes of everything kicking back off.

No joy. My logstash-plain is now filling up with errors:

[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2019.03.18", :_type=>"doc", :routing=>nil}, #LogStash::Event:0x699ac1b], :response=>{"index"=>{"_index"=>"filebeat-2019.03.18", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [default]: No handler for type [string] declared on field [message]", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"No handler for type [string] declared on field [message]"}}}}}

In the reading that I have done, I have come to believe that my template is wonky - maybe it didn't update with the rest of the software? I dunno. I am fuzzy on the template thing and how it all works.

I have seen a couple of posts about looking at the template by using GET localhost:9200/_template/template - but I get nothing. When I call GET localhost:9200/_template, I get a result which is about a mile long.

I have also read an article about setting up a default template with a POST command, but I am not sure even still about what to do.

Can I delete my template and restart logstash to have it build a default template? Should I try changing the template somehow?

My template is too long to post here, so I cna't share it.

Thanks

[Julien]

type string was removed in 5.0 (https://www.elastic.co/blog/strings-are-dead-long-live-strings). However a template in 5.x mentioning type "string" will be working except for the fact it will add some messages in the deprecation logs (if you had used the upgrade assistant in 5.6 it will guide you through the step including deprecation log). If you had old template from v2; the index created using that template will use the new types "text" for analyzed field or keyword for not_analyzed. This won't work from 6.0, creating an index from a template which still mention string will just fail.

So you should probably check if GET _template and look for "string"
...then copy any individual template which has string, and update it with valid definition.
When you try to save a template v6 (or in v5) with invalid definition, you will get an error which you can use to correct any issue till the template definition you send is valid.
Same goes for a template using a deprecated feature, it will save the template in that case but a warning will appear in the response (and on the deprecation logs) which you can correct. I am mentioning this because type string may not be the only deprecated feature you are still using in those templates, if you get deprecated warning in 6.x, correcting those now will avoid you the issue when upgrading to 7.x
Thanks