Elasticsearch 5.x: Is it possible to ignore missing inner object when indexing?

[shangsunset (Alvin)]

So in my mapping shown below, lonlat is geo-shape inside location.

"location": {
  "type": "object",
  "properties" : {
    "lonlat" : {
      "type" : "geo_shape"
     },
     ...
  }
},

My problem is that when indexing documents, some data don't have location and Elasticsearch would return error {"type":"mapper_parsing_exception","reason":"failed to parse [location.lonlat]","caused_by":{"reason":"shape type not included","type":"parse_exception"}} and reject the whole document.

Is there a way to make Elasticsearch ignore the missing properties in location and index the document anyway? I saw there is a new parameter under geo-shape type called ignore-malformed in Elasticsearch 6.x, will that help my situation?

Thanks!

[whicter (Viktor)]

What I did for a similar situation is to check if certain error is in [tags] and do the following:

if "ERROR_TYPE" in [tags] {
        mutate {
            add_field => { "parsing_error" => "ERROR_TYPE" }
        }
    }
}

and create a custom search criteria in kibana to exclude those records. Hope that helps

[shangsunset (Alvin)]

Hi @whicter ,

Thanks for your response. What are [tags]? Are you suggesting I should check for the error type when indexing data and handle the error accordingly? If the error is thrown, doesn't it mean that the Elasticsearch already rejected the document?

Thanks.

[whicter (Viktor)]

I did stdout { codec => rubydebug } together with elasticsearch. If there are some parse error for the single message, it will be put into [tags].

For my scenarios, the message will still be sent to elasticsearch but the grok filter just doesn't work for those messages