Rejecting mapping update to as the final mapping would have more than 1 type: [log, doc]

Elastic Stack Logstash
1

I'm trying to use logstash 6.0.1 and I'm getting the following error from filebeats 6.0.1. I understand the error but not sure how to correct this issue.

Rejecting mapping update to [filebeat-6.0.1-postgres-2017.11.29] as the final mapping would have more than 1 type: [log, doc]

2

I am not specifying the document_type anywhere.

3

Here are my settings .... what am i missing?

cat logstash.yml | grep -v "#"
node.name: p-gp2-es46-4.*****
path.data: /data/logstash
http.host: "p-gp2-es46-4."
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://p-gp2-es46-8.:9200
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: *****

cat pipelines.yml

  • pipeline.id: beats_pipeline
    path.config: "/etc/logstash/pipelines/syslog_pipeline.conf"

cat pipelines/syslog_pipeline.conf
input {
beats {
client_inactivity_timeout => 120
port => 5044
}
}

filter {
}

output {
if "sys_log" in [tags] {
if "_grokparsefailure" in [tags] {
file {
path => "/var/log/logstash/_grokparsefailure/grokparsefailure_sys_log.log"
}
}

elasticsearch {
  hosts => "d-gp2-es46-8.*****:9200"
  manage_template => false
  index => "%{[@metadata][beat]}-%{[@metadata][version]}-sys-log-%{+YYYY.MM.dd}"
}

}
}

4

I took logstash out and have filebeat connecting directly to elasticsearch and still receive the same error. Not sure where the problem lies .... either elasticsearch or filebeats.

2017-12-15T21:34:10Z WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [filebeat-6.0.1-2017.12.15] as the final mapping would have more than 1 type: [log, doc]"}

5

Also filebeats is passing doc for _type so it must be elasticsearch that is some how inserting log in for _type. Any ideas where?

[2017-12-15T21:53:31,894][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-6.0.1-postgres-2017.11.29", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x1e6ed340], :response=>{"index"=>{"_index"=>"filebeat-6.0.1-postgres-2017.11.29", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-6.0.1-postgres-2017.11.29] as the final mapping would have more than 1 type: [log, doc]"}}}}

6

Check if you have any old index template that might be specifying a different type and therefore causing a conflict.

3 Likes
7

I'm not seeing anything. Also this is a fresh install of 6.0.1 so there shouldn't be anything.

8

Do you have an old version of Filebeat somewhere?

9

No, I installed 6.0.1 and only have 1 filebeat setup right now to hit ES 6.0.1.

[root@d-gp2-dbp7-1:~]# /usr/share/filebeat/bin/filebeat --version
filebeat version 6.0.1 (amd64), libbeat 6.0.1

10

I went an installed a new server with a new filebeats install of 6.0.1 and it started to work. Not sure what the difference was but it is working now. So the issues seams to have been with the upgraded filebeats from 5.6.1 to 6.0.1.

11

created a backup file of the filebeat.yml ... tried to make some changes and failed again and then put back the filebeat.yml that did work and failing again. Not sure what is causing this.

12

What would cause it to work one day and the next no?

2017-12-20T18:43:55Z WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [filebeat-6.0.1-2017.12.20] as the final mapping would have more than 1 type: [log, doc]"}

13

You should probably read https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html

The document type used by beats in v6 is doc, I think at some point in v5 it was log (definitely for logstash). So if a v6 beat sends a document into your index the type for your index is set to doc, and anything trying to index a document of type log will get that error.

If the first document in the index is of type log then anything trying to index a document of type doc will get that error. So it sounds like you have two different things indexing documents (logstash and beats perhaps) which are using different types.

If one of the logs infrequently then the problem would go away each day as a new index is created. Then whoever indexes the first document of the day locks the type of the index.

14

I understand the error but just not sure where the "log" document_type could be set. I'm able to go straight from filebeat to elasticsearch now. When going from filebeat -> logstash -> elasticsearch .... it still gives that error.

[2017-12-20T19:50:13,923][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-6.0.1-postgres-2017.11.29", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x2ea6bbce], :response=>{"index"=>{"_index"=>"filebeat-6.0.1-postgres-2017.11.29", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-6.0.1-postgres-2017.11.29] as the final mapping would have more than 1 type: [log, doc]"}}}}

15

Figured it out ... ended up being a few things ... i did have an index template that was referencing "log". Also logstash was off by one letter pointing to the wrong elasticsearch. Seems to be working now.

1 Like
16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.