Elasticsearch 5.x: Is it possible to ignore missing inner object when indexing?


(Alvin) #1

So in my mapping shown below, lonlat is geo-shape inside location.

"location": {
  "type": "object",
  "properties" : {
    "lonlat" : {
      "type" : "geo_shape"
     },
     ...
  }
},

My problem is that when indexing documents, some data don't have location and Elasticsearch would return error {"type":"mapper_parsing_exception","reason":"failed to parse [location.lonlat]","caused_by":{"reason":"shape type not included","type":"parse_exception"}} and reject the whole document.

Is there a way to make Elasticsearch ignore the missing properties in location and index the document anyway? I saw there is a new parameter under geo-shape type called ignore-malformed in Elasticsearch 6.x, will that help my situation?

Thanks!


(Viktor) #2

What I did for a similar situation is to check if certain error is in [tags] and do the following:

if "ERROR_TYPE" in [tags] {
        mutate {
            add_field => { "parsing_error" => "ERROR_TYPE" }
        }
    }
}

and create a custom search criteria in kibana to exclude those records. Hope that helps


(Alvin) #3

Hi @whicter ,

Thanks for your response. What are [tags]? Are you suggesting I should check for the error type when indexing data and handle the error accordingly? If the error is thrown, doesn't it mean that the Elasticsearch already rejected the document?

Thanks.


(Viktor) #4

I did stdout { codec => rubydebug } together with elasticsearch. If there are some parse error for the single message, it will be put into [tags].

For my scenarios, the message will still be sent to elasticsearch but the grok filter just doesn't work for those messages


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.