Kibana Scripted Fields (add fields)

Elastic Stack Kibana
1

I'm looking for some pointers on whether what I am trying to do is possible or if I'm wasting time looking. :slight_smile:

I have multiple winlogbeat sources injecting directly into a winlogbeat index, bypassing my Logstash server. I'm looking to extract the first line of the message from the message field and add that as an additional field. I can do this with a 7 liner in Logstash but I don't have this option for this index.
I can't tell if the scripted field allows the option of adding a field like a mutate would. In Logstash, it would simply be:

filter {
    if "A user account was disabled." in [message] {
    mutate {
        copy => { "message" => "message_header" }
    }
    mutate {
        split => ["message_header", "."]
    }
    mutate {
        replace => [ "message_header", "%{[message_header][0]}" ]
    }
    }

As far as I've got before wondering if it's even possible in a Kibana scripted field is:

 if (doc['message'].value == "A user account was disabled.") {
  return "A user account was disabled.";
 }

This, as far as I can tell, would replace the message field (something I don't wish to do). How can I add an additional field (or copy and mutate the copy as above in Logstash) to get the same result?
I'm cautious about applying this to test as all I have to work with is live data at the mo.

2

Scripted fields configured in your index pattern in Kibana are working like you want them to - they are only adding new fields, not overwriting existing ones. When writing the script you have access to all fields in the current document and can base the value of the scripted field based on those - replicating your logstash pipeline should be easy with painless (the scripting language used by Kibana).

Reference: https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

If you want to test this on live data, but not disturb your users, you could create a separate index pattern and use it for testing. E.g. if your production index pattern is called my-data-indices-*, temporarily create a my-data-indice*pattern - this will match the same documents, but it's a separate saved object so you can play around with the pattern without worrying it will have any impact on the production dashboards.

3

Thanks for the pointer. Do you know if it's possible to check for a value withing a field?
For example:

if (doc['winlog.event_data.Status'].value == "1234567890") {
  return "Unknown";
 }

Would work if the value in the field winlog.event_data.Status was explicitly "1234567890" but how would I test say the message field for a specific string as shown below:

if (doc['winlog.event_data.Status'].contains == "12345") {
  return "Unknown";
 }
4

If you are a Java developer, Painless is very easy to pick up because it's super close to Java. String manipulation works in the same way.

However as this is a super common thing to do there is a special regex syntax in painless you can use here:

if (doc['winlog.event_data.Status'].value ==~ /12345/) {
  return "Unknown";
 }

For more things check out the painless guide - it also has a section about regular expressions:
https://www.elastic.co/guide/en/elasticsearch/painless/7.0/painless-examples.html#modules-scripting-painless-regex

5

So I've just come back to this and spent the last hour tinkering.
I can't see why this wouldn't be valid but the code never evaluates anything so I'm guessing it is. I can run this against other fields with a positive result but not the message field in a Winlogbeat index.

if(doc["message"].value ==~ /An account was successfully logged on./){
return "test"
}

I'm trying to scan all entries into this index and look for the above mentioned text string in the winlogbeat message field. If I find it, I want an additional field with the contents "test" added to the index record.

Is it not possible to do this with the message field?

It's exactly the same problem as discussed here

6

Please check your mapping whether message is a text field - if you haven't enabled doc values specifically for it, the script won't have access to it this way.

You can either enable doc values for the field or add a store: true to it and access it via _fields.

The Elasticsearch documentation does a great job explaining it: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/modules-scripting-fields.html#modules-scripting-doc-vals and https://www.elastic.co/guide/en/elasticsearch/reference/7.6/modules-scripting-fields.html#modules-scripting-stored

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.