Logstash 6.0 breaks winlogbeat agent indexing

Elastic Stack Logstash
1

I've got a winlogbeat agents version 5.x sending data to a kafka queue with a topic name of winlogbeat.

Our indexer cluster is pulling out of that topic. With 6.0 Elasticsearch and 5.6.4 agents / logstash 5.6.4 all is good.

As soon as I upgrade logstash to 6.0 winlogbeat indexing stops with this error.

[2017-11-20T20:27:06,639][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{[@metadata][index]}-2017.11.21", :_type=>"wineventlog", :_routing=>nil}, #<LogStash::Event:0x44a2d980>], :response=>{"index"=>{"_index"=>"%{[@metadata][index]}-2017.11.21", "_type"=>"wineventlog", "_id"=>"grZm3F8BaYAZqFFMeCqC", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [%{[@metadata][index]}-2017.11.21] as the final mapping would have more than 1 type: [metricsets, wineventlog]"}}}}

The other 5.4.6 logstash instances still work fine sending data to elasticsearch 6.0

Is this a problem with the template in elasitcsearch still being 5.6.4 and not 6.0?

In all honesty I don't understand what this error indicates or the steps needed to narrow it down and fix.

I've read about the removal of the multiply mapping types but I don't see why just updating logstash from 5.6.4 to 6.0 would cause this.

2

See the following topics:

3

We ended up dealing with this issue by forcing type to be "doc" on all new indices in our logstash configuration. This should work well and help us stop using type in preparation for its eventual demise.

It took us a little while to understand that the issue was the fact that in data coming in had various values for type. Once we understand that it was fairly easy to fix.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.