Use special (null?) value for missing field

Elastic Stack Elasticsearch
1

I want to import some data into ES.

There is an integer field in my template and I want to use strict mapping.

However, there might be the case that in some documents, the specific filed is missing.

What I want to achieve is to have a special value, say null (not the string "null" !) for the documents where the field is missing.

If I perform

  if !([my_field]) {
      mutate {
        add_field => [ "[my_field]", null ]
      }
    }

... and the type is set to integer in my index template, it will convert it to '0' during import;

Is there a way to keep a special value for missing fields during import?

Or do I have to resort to just assigning a special pre-defined value by convention (e.g. -999)

2

I have managed to do this via the following inline ruby code:

if !([my_field]) {
  ruby {
    code => "event.set('my_field' , nil)"
  }

However the problem now is that the following query:

"query": {
    "match" : {
        "my_field" : null
    }
}

will fetch ALL documents and not only those whose specific field has the value null .

How can I prevent this behavior?

3

If you are planning to fetch the fields that have missing field, then you could use the below query

"query": {
    "bool": {
        "must_not": {
            "exists": {
                "field": "my_field"
            }
        }
    }
}

Hope this helps ..!!