This issue affects Logstash 6.0.0 users indexing to Elasticsearch 6.0 when multiple “type” field values are present in Logstash events. It affects both new users on Logstash 6.0.0, and existing users upgrading to Logstash 6.0.0. Common scenarios include:
- Receiving data from multiple types of Beats (i.e. Filebeat, Metricbeat)
- Receiving data from Filebeat tailing multiple files with different types
- Multiple inputs that tag different types by data source
- Tagging of multiple types in your pipeline with conditionals
The Problem
The bundled Elasticsearch output plugin v9.0.0 in Logstash 6.0.0 will use the “type” field value, if defined, in the event to set the document type when indexing to Elasticsearch 6.0. In the scenarios enumerated above, users may encounter an error for trying to index multiple types to the same index. Example below:
[2017-11-21T14:26:01,991][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :response=>{"error"=>{"reason"=>"Rejecting mapping update to [myindex] as the final mapping would have more than 1 type: [type1, type2]"}}}}
The Solution
Logstash 6.0.0 users must make sure to take two steps of corrective action:
-
Explicitly set this parameter in the Elasticsearch output section of the Logstash configuration file:
document_type => doc -
Either write to a new index, or clear the existing index being written to before continued processing.
With this parameter set, Logstash 6.0.0 will now effectively be able to index into Elasticsearch 6.0+. This fix will be included directly in Logstash 6.0.1, and also backported to Logstash 5.6.