Logstash 6.0 breaks winlogbeat agent indexing

[kwslavens (Kelly Slavens)]

I've got a winlogbeat agents version 5.x sending data to a kafka queue with a topic name of winlogbeat.

Our indexer cluster is pulling out of that topic. With 6.0 Elasticsearch and 5.6.4 agents / logstash 5.6.4 all is good.

As soon as I upgrade logstash to 6.0 winlogbeat indexing stops with this error.

[2017-11-20T20:27:06,639][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{[@metadata][index]}-2017.11.21", :_type=>"wineventlog", :_routing=>nil}, #<LogStash::Event:0x44a2d980>], :response=>{"index"=>{"_index"=>"%{[@metadata][index]}-2017.11.21", "_type"=>"wineventlog", "_id"=>"grZm3F8BaYAZqFFMeCqC", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [%{[@metadata][index]}-2017.11.21] as the final mapping would have more than 1 type: [metricsets, wineventlog]"}}}}

The other 5.4.6 logstash instances still work fine sending data to elasticsearch 6.0

Is this a problem with the template in elasitcsearch still being 5.6.4 and not 6.0?

In all honesty I don't understand what this error indicates or the steps needed to narrow it down and fix.

I've read about the removal of the multiply mapping types but I don't see why just updating logstash from 5.6.4 to 6.0 would cause this.

[magnusbaeck (Magnus Bäck)]

See the following topics:

Problem with multiple mapping types in 6.0. Not sure why

I have previously commented on another topic with a similar issue, but as that was beats related, and fixed by updates there, and mine is not, i a, creating this topic. I am getting errors due to multiple mapping types, as per the 6.0 breaking changes, but i don't know why i have multiple mapping types. The data is delivered by a powershell script which gathers performance counter data and sends it to logstash. My logstash configuration looks like this: input { tcp { port => 5560 codec => …

Help please.. breaking changes?

Upgrading to 6.0.0 of the ELK stack Can anyone please help me why i get this entrys over and over in the Logstash log [2017-11-16T20:49:36,973][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-2017.11.16", :_type=>"wineventlog", :_routing=>nil}, #LogStash::Event:0x59ffd445], :response=>{"index"=>{"_index"=>"winlogbeat-2017.11.16", "_type"=>"wineventlog", "_id"=>"LcRhxl8BA5Maqd7NKABD", "status"=>40…

[kwslavens (Kelly Slavens)]

We ended up dealing with this issue by forcing type to be "doc" on all new indices in our logstash configuration. This should work well and help us stop using type in preparation for its eventual demise.

It took us a little while to understand that the issue was the fact that in data coming in had various values for type. Once we understand that it was fairly easy to fix.