KQL and scripted fields

[thomasv95 (Thomas Vermaelen)]

Hello, I've been having issues with KQL queries using scripted fields. Specifically, queries on numerical fields using operators such as "<=" or ">" throw the following error when using a Lens : [lens_merge_tables] > [esaggs] > EsError

It looks like I get a more detailed error using an Aggregation (see attached screenshot).

Interestingly enough, the KQL operator ":" works just fine.

All I am doing in the scripted field is multiplying a numerical field by 2, and there are no missing values.

I am at a complete loss since the Kibana documentation clearly states that all KQL operators are compatible with scripted fields.

I absolutely need to use KQL filtering since I'm using a Filter Ratio in the TSVB tool.

Your support would be much appreciated. Thank you

[Marius_Dragomir]

hello,

This seems like a bug indeed. If you're using ES > 7.11 you can solve this issue by changing from scripted fields to runtime fields.
For example, if you had a scripted field like this:
doc['FlightDelayMin'].value * 2

you can change it to a runtime field from the Dev Tools

PUT kibana_sample_data_flights/_mapping
{
    "runtime": {
      "minz": {
        "type": "long",
        "script": {
          "source": "emit(doc['FlightDelayMin'].value * 2)"
        }
      }
    }
}

where kibana_sample_data_flights is to be replaced with your index pattern name and minz with your field name.

[Marius_Dragomir]

And this is the issue for the scripted fields error: [kql] Creating filters with numeric scripted fields sends incorrect type · Issue #98761 · elastic/kibana · GitHub

[thomasv95 (Thomas Vermaelen)]

Hi Marius,

Thank you for your detailed response, using runtime fields did indeed provide a quick fix to my problem. I just saw this morning that a PR was made to solve this bug in Kibana (if I'm not mistaken). My team and I hope that it can be resolved ASAP as we are scheduled to deliver our prototype in the upcoming weeks to be used by non-technical end users who would prefer using scripted fields rather than runtime fields.

Thank you again for your time and support.

[Marius_Dragomir]

Just to manage the expectations, that PR is set to go in 7.14.0 which is still some time away. Right now 7.13.0 is close to being released in a couple weeks and that fix won't get in it due to code freeze. I wouldn't expect 7.14.0 until mid summer

[thomasv95 (Thomas Vermaelen)]

Ok, thanks for letting me know, I will pass the message on to my team.