Returning 'null' or nothing from scripted fields

Elastic Stack Kibana
1

Hello,

I'd like to be able to return a 'null' from a scripted field defined in Kibana - e.g.,

my_scripted_field = doc['field_1'].value != 0 ? doc['field_2'].value : null

The idea is that if field_1 doesn't exist (== 0 in Lucence expressions), then there is no value assigned to my_scripted_field - as if the field doesn't exist in a document.

The ultimate use-case, is to be able to run a 'count distinct' (cardinality) aggregation on my_scripted_field, but without it counting 0 as a distinct value (which would be the case if I allow the ternary operator to revert to default behaviour when == 0).

Is there any way to return a 'null' to get the desired behaviour with Lucene expressions, and if not, is this something I could achieve by enabling Groovy scripting?

2

As far as I know, Lucence expressions only support numbers, so I don't think it's possible to do what you want that way. Groovy scripting should allow it though, since it supports multiple values types, but you might run in to an issue with the type mappings when you do your aggregation. That is, if Java is expecting a value to be a number, but instead it's null, it may error instead of working the way you expect.

You'd have to try it out and see I guess. You could also try asking over in the Elasticsearch section, they're bound to be more knowledgeable than I am about Lucence expressions and the type handling in aggregations. There may be another way to do what you need without using Groovy too.

3

Thanks Joe. I can't return a null using Groovy either (results in a nested ClassCastException).

I'll explore this use-case on the Elasticsearch section like you suggest.

4

UPDATE:

While composing an example in Sense of what I'm trying to achieve, I realised I could achieve what I want with the following type of Groovy Script - e.g.,

i.e. if I don't specify a return value for the 'else case', then the field isn't assigned a value and doesn't contribute anything to a cardinality aggregation.

However, when I try to get this operational in Kibana using,

{"script": "if (doc['field_1'].value != 0) doc['field_2'].value", "lang": "groovy"} ,

in the JSON Input for the metric, get an error in Kibana - e.g.,

Screen Shot 2016-07-28 at 12.58.38.png2870×638 175 KB

If I copy and paste the 'Elasticsearch Resquest' of the visualisation directly into Sense (as a sanity check), I notice that the "field" is being set when it should be missing (redundant) - e.g.,

... 
  ,
  "aggs": {
    "1": {
      "cardinality": {
        "field": "a_field_kibana_forces_me_to_select_in_UI",
        "script": "if (doc['field_1'].value != 0) doc['field_2'].value",
        "lang": "groovy"
      }
    }
  }
}

Bug in Kibana or am I doing something wrong in the UI?

5

Is field_1 not the field you are selecting in the UI for the aggregation?

I think the JSON input is merged on top of what you choose in the interface, which means you can override the field you choose in the UI - but I could be wrong about that. You can try it out by adding (I think) "field": "a_suitable_field" to the JSON input.

I think the intention is that you add a Scripted Field to the index pattern in the settings interface, but I'm not entirely sure if you can, or how you would do that with a groovy script - everything in the documentation seems to be limited to Lucene Expressions. I thought there was a way to use scripts from files in Kibana, and that doing so was a requirement to use Groovy scripts, but I can't find any info on it now.

Also, it's a bit of a tangent, but I believe that Groovy scripting was completely removed in ES 2.0, and 5.0 is bringing its own "painless" scripting language, which Kibana will eventually have native support for. Not sure it that'll happen in time for the 5.0 release of Kibana or not though.