Ignore_malformed within an object

Elastic Stack Elasticsearch
1

Hi, can anyone provide confirmation or clarification if using ignore_malformed on a property within an object is supported.

The docs say they are not supported on object types but does this extend to properties within the object?

for example :

"dynamic_templates": [
        {
          "custom_date_iso": {
            "path_match": "custom.*",
            "match_mapping_type": "date",
            "mapping": {
              "format": "strict_date_optional_time",
              "ignore_malformed": true,
              "type": "date"
            }
          }
        }
]

with the above dynamic mapping, when I submit a doc

{
  "custom.date": "1970-01-01"
}

it results in:

 "properties": {
        "custom": {
          "properties": {
            "date": {
              "type": "date",
              "format": "strict_date_optional_time",
              "ignore_malformed": true
            }
          }
        }
      }

This allows me to send bad data for "custom.date" and have it ignored which is what I want.

Is this a supported use case for ignore_malformed?

Thanks

2

Hi @TiredPanda

I think it's about which type ignore_malformed is supported too.

The doc says that for the Nested data type, Object data type, Range data types it is not supported.

But for is supported for types: Numeric, Boolean, Date, Date nanoseconds, Geopoint, Geoshape, IP.
So if you test for a text or keyword type you will see that it is not supported.

1 Like
3

This means that the ignore_malformed will not work for object fields.

For example, if you send this to elasticsearch:

{
    "custom": "some value"
}

And then, try to send this:

{
    "custom" : {
        "nested": "some value"
    }
}

Then the second document will be rejected completely, regardless if you have ignore_malformed configured.

The case you mentioned is different, because you would have something like this:

{
    "custom": {
        "date": "1970-01-01"
    }
}

and

{
    "custom": {
        "date": "this-is-not-a-valid-date"
    }
}

In this case, only the custom.date field would be rejected, not the entire document.

1 Like