Could not index event to Elasticsearch. Root mapping definition has unsupported parameters

Elastic Stack Logstash
1

Hi,

I have recently upgraded from 6.6 to 7.6.2 and it all went smoothly enough. However I checked my logs this morning and have found the following:

[2020-04-15T15:50:50,851][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-qa01-stats-2020.04.15", :_type=>"collectd", :_routing=>nil}, 2020-04-15T14:03:42.347Z qa01grn01.redmatter.com %{message}], :response=>{"index"=>{"_index"=>"logstash-qa01-stats-2020.04.15", "_type"=>"collectd", "_id"=>nil, "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [_default_]: Root mapping definition has unsupported parameters: [_all : {norms=false, enabled=false}]", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"Root mapping definition has unsupported parameters: [_all : {norms=false, enabled=false}]"}}}}}

Here is my full template:

    {
      "template" : "logstash-qa01-stats-*",
      "settings" : {
        "index.refresh_interval" : "10s"
      },
      "mappings" : {
        "_default_" : {
           "_all" : { "enabled" : false, "norms" : false },
          "dynamic_templates" : [ {
              "string_fields": {
                "match": "*",
                "match_mapping_type": "string",
                "mapping": { "type": "string", "index": true, "doc_values": true }
              }
            }, {
              "byte_fields" : {
                "match" : "*",
                "match_mapping_type" : "long",
                "mapping" : { "type" : "double", "doc_values": true }
              }
            }, {
              "short_fields" : {
                "match" : "*",
                "match_mapping_type" : "long",
                "mapping" : { "type" : "double", "doc_values": true }
              }
            }, {
              "integer_fields" : {
                "match" : "*",
                "match_mapping_type" : "long",
                "mapping" : { "type" : "double", "doc_values": true }
              }
            }, {
              "long_fields" : {
                "match" : "*",
                "match_mapping_type" : "long",
                "mapping" : { "type" : "double", "doc_values": true }
              }
            }, {
              "float_fields": {
                "match": "*",
                "match_mapping_type": "double",
                "mapping": { "type": "double", "doc_values": true }
              }
            }],
            "properties" : {
              "@timestamp": { "type" : "date", "format": "date_optional_time" },
              "@version" : { "type" : "keyword", "index": true },
              "type_instance" : { "type" : "text", "index" : true, "norms" : false,
                "fields" : {
                  "raw" : { "type": "text", "index" : true }
                }
              }
            }
        }
      }
    }

I can't quite work out which bit is causing an issue. Does anyone have any suggestions?

2

The _all meta field was removed in 7.0 after being deprecated in 6.0.

3

Thanks Badger,

I have tried removing the _all field and reapplying the template via the API Console:

 {
    	"template": "logstash-qa01-stats-*",
    	"settings": {
    		"index.refresh_interval": "10s"
    	},
    	"mappings": {
    		"_default_": {
    			"dynamic_templates": [{
    				"string_fields": {
    					"match": "*",
    					"match_mapping_type": "string",
    					"mapping": {
    						"type": "string",
    						"index": true,
    						"doc_values": true
    					}
    				}
    			}, {
    				"byte_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"short_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"integer_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"long_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"float_fields": {
    					"match": "*",
    					"match_mapping_type": "double",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}],
    			"properties": {
    				"@timestamp": {
    					"type": "date",
    					"format": "date_optional_time"
    				},
    				"@version": {
    					"type": "keyword",
    					"index": true
    				},
    				"type_instance": {
    					"type": "text",
    					"index": true,
    					"norms": false,
    					"fields": {
    						"raw": {
    							"type": "text",
    							"index": true
    						}
    					}
    				}
    			}
    		}
    	}
    }

But I am getting the following error:

{
  "status": 400,
  "error": {
    "caused_by": {
      "reason": "Root mapping definition has unsupported parameters:  [_default_ : {dynamic_templates=[{string_fields={mapping={index=true, type=string, doc_values=true}, match_mapping_type=string, match=*}}, {byte_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {short_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {integer_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {long_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {float_fields={mapping={type=double, doc_values=true}, match_mapping_type=double, match=*}}], properties={@timestamp={format=date_optional_time, type=date}, type_instance={norms=false, index=true, type=text, fields={raw={index=true, type=text}}}, @version={index=true, type=keyword}}}]",
      "type": "mapper_parsing_exception"
    },
    "root_cause": [
      {
        "reason": "Root mapping definition has unsupported parameters:  [_default_ : {dynamic_templates=[{string_fields={mapping={index=true, type=string, doc_values=true}, match_mapping_type=string, match=*}}, {byte_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {short_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {integer_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {long_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {float_fields={mapping={type=double, doc_values=true}, match_mapping_type=double, match=*}}], properties={@timestamp={format=date_optional_time, type=date}, type_instance={norms=false, index=true, type=text, fields={raw={index=true, type=text}}}, @version={index=true, type=keyword}}}]",
        "type": "mapper_parsing_exception"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [_default_ : {dynamic_templates=[{string_fields={mapping={index=true, type=string, doc_values=true}, match_mapping_type=string, match=*}}, {byte_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {short_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {integer_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {long_fields={mapping={type=double, doc_values=true}, match_mapping_type=long, match=*}}, {float_fields={mapping={type=double, doc_values=true}, match_mapping_type=double, match=*}}], properties={@timestamp={format=date_optional_time, type=date}, type_instance={norms=false, index=true, type=text, fields={raw={index=true, type=text}}}, @version={index=true, type=keyword}}}]"
  }
}

If I change my template to:

 {
    	"template": "logstash-qa01-stats-*",
    	"settings": {
    		"index.refresh_interval": "10s"
    	},
    	"mappings": {
    			"dynamic_templates": [{
    				"string_fields": {
    					"match": "*",
    					"match_mapping_type": "string",
    					"mapping": {
    						"type": "string",
    						"index": true,
    						"doc_values": true
    					}
    				}
    			}, {
    				"byte_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"short_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"integer_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"long_fields": {
    					"match": "*",
    					"match_mapping_type": "long",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}, {
    				"float_fields": {
    					"match": "*",
    					"match_mapping_type": "double",
    					"mapping": {
    						"type": "double",
    						"doc_values": true
    					}
    				}
    			}],
    			"properties": {
    				"@timestamp": {
    					"type": "date",
    					"format": "date_optional_time"
    				},
    				"@version": {
    					"type": "keyword",
    					"index": true
    				},
    				"type_instance": {
    					"type": "text",
    					"index": true,
    					"norms": false,
    					"fields": {
    						"raw": {
    							"type": "text",
    							"index": true
    						}
    					}
    				}
    			}
    		}
    	}

Elastic accepts it, but then my logstash starts complaining with the following:

[2020-04-15T18:03:42,481][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-04-15T18:03:43,789][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out {:url=>https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/, :error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

Any ideas what might be happening here?

4

The _default_ mapping was also removed in 7.0. I would take your template over to the elasticsearch forum and ask how it needs to be updated to work with 7.0

5

Thanks Badger,

I have removed the depreciated fields but now my whole logstash/elasticsearch cluster is broken:

[2020-04-15T18:03:42,481][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-04-15T18:03:43,789][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out {:url=>https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/, :error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.