Rejecting mapping update to as the final mapping would have more than 1 type: [log, doc]


(Kyle Stephenson) #1

I'm trying to use logstash 6.0.1 and I'm getting the following error from filebeats 6.0.1. I understand the error but not sure how to correct this issue.

Rejecting mapping update to [filebeat-6.0.1-postgres-2017.11.29] as the final mapping would have more than 1 type: [log, doc]


(Kyle Stephenson) #2

I am not specifying the document_type anywhere.


(Kyle Stephenson) #3

Here are my settings .... what am i missing?

cat logstash.yml | grep -v "#"
node.name: p-gp2-es46-4.*****
path.data: /data/logstash
http.host: "p-gp2-es46-4."
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://p-gp2-es46-8.
:9200
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: *****

cat pipelines.yml

  • pipeline.id: beats_pipeline
    path.config: "/etc/logstash/pipelines/syslog_pipeline.conf"

cat pipelines/syslog_pipeline.conf
input {
beats {
client_inactivity_timeout => 120
port => 5044
}
}

filter {
}

output {
if "sys_log" in [tags] {
if "_grokparsefailure" in [tags] {
file {
path => "/var/log/logstash/_grokparsefailure/grokparsefailure_sys_log.log"
}
}

elasticsearch {
  hosts => "d-gp2-es46-8.*****:9200"
  manage_template => false
  index => "%{[@metadata][beat]}-%{[@metadata][version]}-sys-log-%{+YYYY.MM.dd}"
}

}
}


(Kyle Stephenson) #4

I took logstash out and have filebeat connecting directly to elasticsearch and still receive the same error. Not sure where the problem lies .... either elasticsearch or filebeats.

2017-12-15T21:34:10Z WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [filebeat-6.0.1-2017.12.15] as the final mapping would have more than 1 type: [log, doc]"}


(Kyle Stephenson) #5

Also filebeats is passing doc for _type so it must be elasticsearch that is some how inserting log in for _type. Any ideas where?

[2017-12-15T21:53:31,894][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-6.0.1-postgres-2017.11.29", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x1e6ed340], :response=>{"index"=>{"_index"=>"filebeat-6.0.1-postgres-2017.11.29", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-6.0.1-postgres-2017.11.29] as the final mapping would have more than 1 type: [log, doc]"}}}}


(Christian Dahlqvist) #6

Check if you have any old index template that might be specifying a different type and therefore causing a conflict.


(Kyle Stephenson) #7

I'm not seeing anything. Also this is a fresh install of 6.0.1 so there shouldn't be anything.


(Christian Dahlqvist) #8

Do you have an old version of Filebeat somewhere?


(Kyle Stephenson) #9

No, I installed 6.0.1 and only have 1 filebeat setup right now to hit ES 6.0.1.

[root@d-gp2-dbp7-1:~]# /usr/share/filebeat/bin/filebeat --version
filebeat version 6.0.1 (amd64), libbeat 6.0.1


(Kyle Stephenson) #10

I went an installed a new server with a new filebeats install of 6.0.1 and it started to work. Not sure what the difference was but it is working now. So the issues seams to have been with the upgraded filebeats from 5.6.1 to 6.0.1.


(Kyle Stephenson) #11

created a backup file of the filebeat.yml ... tried to make some changes and failed again and then put back the filebeat.yml that did work and failing again. Not sure what is causing this.


(Kyle Stephenson) #12

What would cause it to work one day and the next no?

2017-12-20T18:43:55Z WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [filebeat-6.0.1-2017.12.20] as the final mapping would have more than 1 type: [log, doc]"}


#13

You should probably read https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html

The document type used by beats in v6 is doc, I think at some point in v5 it was log (definitely for logstash). So if a v6 beat sends a document into your index the type for your index is set to doc, and anything trying to index a document of type log will get that error.

If the first document in the index is of type log then anything trying to index a document of type doc will get that error. So it sounds like you have two different things indexing documents (logstash and beats perhaps) which are using different types.

If one of the logs infrequently then the problem would go away each day as a new index is created. Then whoever indexes the first document of the day locks the type of the index.


(Kyle Stephenson) #14

I understand the error but just not sure where the "log" document_type could be set. I'm able to go straight from filebeat to elasticsearch now. When going from filebeat -> logstash -> elasticsearch .... it still gives that error.

[2017-12-20T19:50:13,923][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-6.0.1-postgres-2017.11.29", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x2ea6bbce], :response=>{"index"=>{"_index"=>"filebeat-6.0.1-postgres-2017.11.29", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-6.0.1-postgres-2017.11.29] as the final mapping would have more than 1 type: [log, doc]"}}}}


(Kyle Stephenson) #15

Figured it out ... ended up being a few things ... i did have an index template that was referencing "log". Also logstash was off by one letter pointing to the wrong elasticsearch. Seems to be working now.


(system) #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.