Elasticsearch - Kibana - Monitors extraction query fails when documents doesn't have the field

[prashu421 (Prashanth Sripathi)]

I have some monitors setup in Elasticsearch Kibana, using an extraction query to fetch the records where griefAttribute > 0, this query works fine in case all the documents have the said field/attributes, otherwise the monitor fails with below error -

Error -

Error:
Failed fetching inputs:
Failed to execute phase [query], all shards failed;
shardFailures
{
[XayC4jDFF56gYupQ_LdHQ][idx-execution][0]: RemoteTransportException[[62a21ef10083566rgfg4afj163a33e8c69514][<IP_ADDRESS>][indices:data/read/search[phase/query]]];
nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]];
nested: ScriptException[runtime error];
nested: IllegalStateException[A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!];
};
ScriptException[runtime error];
nested: IllegalStateException[A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!];;
java.lang.IllegalStateException: A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!

Query -

{
    "size": 100,
    "query": {
        "bool": {
            "must": [
                {
                    "script": {
                        "script": {
                            "source": "doc['griefAttribute'].value > 0",
                            "lang": "painless"
                        }
                    }
                },
                {
                    "range": {
                        "CreatedOn": {
                            "from": "now-1d/d",
                            "to": "now/d",
                            "include_upper": false
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    }
}

Would it be possible for me omit this field existence check error? incorporating doc[].size()==0 condition, something like this?

//Below is a stupid try, that didn't work
{
  "script": {
    "script": {
      "source": "if (doc[griefAttribute].size() > 0) { doc['griefAttribute'].value > 0 }",
      "lang": "painless"
    },
    "boost": 1
  }
}

Thank you for taking a look at this.

Best
Prashanth Sripathi

[LeeDr (Lee Drengenberg)]

I think you're pretty close. I'm not sure about your case, but I loaded 2 docs in an index like this;

POST discuss/_doc
{
  "griefAttribute": 5
}

POST discuss/_doc
{
  "otherAttribute": 6
}

Then I created an index pattern. And then a scripted field with type number. For this, this script works. One doc returns the griefAttribute 5, and the other returns 99.

if (doc['griefAttribute'].size()==0) { 
    return 99;  
} else {
    return doc['griefAttribute'].value;
}

[prashu421 (Prashanth Sripathi)]

Hello @LeeDr , thanks for getting back to me. I was away on vacation and didn't get a chance to respond back.

Theif..else did not work in the extraction query, but following your example, I typed everything in, and the below statement worked! which is not much different from my initial try -

"source": "if (doc['griefAttribute'].size() > 0) {doc['griefAttribute'].value > 0}"

I don't understand what changed, only difference I see are the space, but the query certainly works, and I will take that for now.

Thank you so much for taking time to look at this.

Best
Prashanth Sripathi