Visualisation Painless script between two characters

[georgejolliffe]

Hi guys,

I have this example string in a field called querystring:

utm_medium=inapp&utm_source=crm&utm_campaign=newsl&utm_term=fishandchips

I am trying to visualise occurrences of all the possible "utm_source". I can't enable Regex on this cluster because it's a managed service outside of ELK directly. Is it possible to use painless, like my example below, to match everything between "utm_source" and the following "&"?

{
"script": "( _value.indexOf('utm_source') > 0 ? _value.substring(0, _value.indexOf('&')) : _value )"
}

I am trying to use this in the advanced JSON element in a visualisation. This is a temporary query on historic data so fixing the parsing isn't really a good solution for me. The position of "utm_source" is not consistent.

Many thanks,

George

[matw (Matthias Wilhelm)]

Hi
If you use Kibana 7.12 you can do that by extending your index pattern with a field!

I've made this work with the following code:

Then you can use this field like any other fields (knowing it's a runtime field, calculated when the query is executed)
If you have an older Kibana you could do similar stuff with scripted fields
Best,
Matthias

[georgejolliffe]

Hi @matw Thanks for your reply and that is a great suggestion.

Unfortunately the only option I have presently is using the Advanced JSON inside of the visualisation.

We are using a managed ELK service through Logz.io and they are not yet running 7.12 and even if they did, i'm not sure if they would expose this capability or not.

Is there any chance your script can be persuaded to work inside of the visualisation?

George

[matw (Matthias Wilhelm)]

Why is using a scripted field no option?

Scripted fields | Kibana Guide [7.12] | Elastic

Best,
Matthias

[georgejolliffe]

Old article but it still stands:

Scripted Fields

[matw (Matthias Wilhelm)]

any field calculated when the query is submitted has worse performance. So scripted fields are disabled on logz.io? here's a discuss issue where someone succeeded using the JSON input, didn't test:

JSON Input vs. Scripted Fields

Hello, I have a requirement to get the average of the sum of two fields in all document and visualize it, something like this: avg("field 1" + "field 2") I have found two ways to do this and curious what the difference is both in terms of "best practices" or performance: Method 1: Create a "scripted field" for the index which would be the SUM of the two required fields stored into a new field (called sum_field1_field2) and get the average of "sum_field1_field2 in metric. Method 2: Create a …

However all approaches have this performance drawback.
Best,
Matthias

[georgejolliffe]

Yea i realise the performance impact is present regardless. I guess their logic is that it's easier to cause 'more slow' when the scripted field is present over a handful of visualisations, but yea, scripted fields are disabled on Logz.io.

I have had some success using manipulating using the visualisation based on this guide but where I run out of talent is programming the logic which reads something like "from utm_source to the next &". Assuming it's even possible.

I appreciate your suggestions though.

[matw (Matthias Wilhelm)]

well it should be similar to the code I wrote in the runtime field editor