Handling sparse data in scripts

(BradVido) #1

Is there any way to check existence of a field within a script, and return null/undefined instead of a value?

Let's say I have a simple groovy script that sums the value of two fields:

doc['field1'].value + doc['field2'].value

Then I run a Stats aggregation in ES and get the average of the script values.

My question is how to handle sparse data, especially when both field1 and field2 are missing. I understand the script would return 0 in this case, which is not correct. The value is actually _missing_ and shouldn't be included in the average. Returning 0 throws the average way off.

(Nik Everett) #2

Would a null check work for you? Maybe the elvis operator if this is groovy.

(BradVido) #3

No, inside the script the value isn't null (it's 0) if it's missing from the document.
I'm curious if I can return null from the script to indicate there is no value, but the problem is I don't really have a way to determine that because missing values are converted to zeroes when the script tries to access them.

(system) #4