Kibana 5 | apply regex on scripted fields

Elastic Stack Kibana
1

Hello,

I'm using Kibana to view my logs.

I have some URLs like this:
/myapp/bla/bla/test.shtml;jsessionid=3CF74CA6abht48753DA243C5338222EE.jvmid?show=blaa

At the moment, I'm using the following regex in logstash to extract the first part of the URL:

(?'request_noparms'[a-zA-Z0-9_.\/]*(?=\b[\;\?\&]|.*))

Result: /myapp/bla/bla/test.shtml

I would like to remove the regex from logstash and to apply it into Kibana, by using a scripted field.

Is it possible to do such a thing ?

Thank you for your help.

Best regards

2

Yes it is, if you're using painless you'll have to enable regular expressions which comes with potential performance caveats, see https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-painless.html#modules-scripting-painless-regex.
Your script would look something like:

Matcher m = /[a-zA-Z0-9_.\/]*(?=\b[\;\?\&]|.*)/.matcher(doc['my_url_field'].value);
if(m.find()) {
  return m.group();
}
return "";

edit: added an empty string return at the bottom to have an explicit output

3

Hello,

Thank you for your help. It works just great !
But now I have another issue.

I have created a scripted field "testfield1" with this painless script:

Matcher m = /\/(?!.*\/)([a-zA-Z0-9_.]*(?=\b[\;\?\&]|.*))/.matcher(doc['request'].value);
if(m.find()) {
  return m.group(1);
}

It works when I add this field into a Kibana vizualisation, but it fails when I try to query the field with this error:

Please see at: http://pastebin.com/jihnBssg

Any idea ?
Thank you for your help

Best regards
Jérôme

4

There are several issues related to scripted fields including this one which is fixed in 5.1.0 (NOT RELEASED YET);

The larger search result;

But there may also be another way to get your desired scripted field result that doesn't have the error. Instead of using a regular expression you could use some other string methods.

In my example I have some URLs in a field named referer and if I want to just get the http or https part off the front of the URL I can use this scripted field;

doc['referer'].value.substring(0, doc['referer'].value.indexOf(":"))
This gets the substring starting at 0 and up to the ":". You could try this in your case and see if it works for you.
I'm not sure how it handles the case where the ":" isn't found as all my data has it.

Regards,
Lee

5

Sorry, after re-reading your post and thinking about it a bit, I don't think you can ever use scripted fields in the query bar in Kibana as that goes directly to Elasticsearch which doesn't know about them. But you should be able to filter on them and use them in aggregations in Visualizations.

6

Hi,
OK, thanks for your help.

Best regards
Jérôme

7

FYI - There is now a blog published on Painless in Kibana scripted fields: https://www.elastic.co/blog/using-painless-kibana-scripted-fields

Make sure you are using Kibana 5.1.1, since prior to that version there were issues with filtering and sorting on some of the more advanced scripted fields. Also make sure that every execution path in your script has a well-defined return statement, otherwise some sorts will still fail. Here is the part of the blog that explains that:

Screen Shot 2016-12-13 at 12.19.04 PM.png1440×662 81.5 KB

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.