Cannot search for events based on scripted fields

[vivekkarne (Vivek Vardhan Reddy)]

Hi,
When i search using scripted fields in kibana, i either get no matching hits or I get a shards failed error. Can anyone please help me to look into this. Entire ELK stack is 7.4.2.

s_query body:

    if(doc['querybody'].size() > 0) {
        String querybody=doc['querybody'].value;
        querybody=/[<>=] {1,}[0-9][^ )]*/.matcher(querybody).replaceAll("= ?");
        querybody=/[<>=] {1,}['][^']*'/.matcher(querybody).replaceAll("= ?");
        querybody=/ts_c_[^\.]*/.matcher(querybody).replaceAll("ts_c_?");
        querybody=/([a-z_]{1,})00[0-9]{2}/.matcher(querybody).replaceAll("\\1");
        querybody=/ IN {1,}\([^)]{1,}\)*/.matcher(querybody).replaceAll(" IN ?");
        querybody=/ LIKE {1,}['][^']*'/.matcher(querybody).replaceAll(" LIKE ?");    
        return querybody;
    }

Getting shards failed:

s_querytimes:

if(doc['querytimems'].size() > 0) { return doc['querytimems'].value/1000.0; }

Regards,
Vivek

[flash1293 (Joe Reuter)]

When you are using a scripted field as comparison in KQL, it always needs to return a numeric value. Add an else condition and return -1 or something similar in there, then it should work

[vivekkarne (Vivek Vardhan Reddy)]

The scripted field will exist only when doc['querytimems'] field is present, else the scripted field will not be present. So writing an else condition will cause all the documents which does not have doc['querytimems'] field to contain the scripted field with value -1. Is this the expected behaviour?

[vivekkarne (Vivek Vardhan Reddy)]

And also can we perform text searches on scripted fields?

[flash1293 (Joe Reuter)]

The scripted field will exist only when doc['querytimems'] field is present, else the scripted field will not be present. So writing an else condition will cause all the documents which does not have doc['querytimems'] field to contain the scripted field with value -1. Is this the expected behaviour?

Yes, this would be necessary to use the scripted field in this way.

And also can we perform text searches on scripted fields?

On scripted fields returning strings you can do text searches as well