Kibana Index - Splitting Index Values and Adding to New Field

Elastic Stack Logstash
1

Hello,

I am trying to figure out how to split the value of an index field and add that to a new field. Basically what I have on Kibana is as shown:

image

This is the json document output of the index fields:

image

I am trying to remove the value "Action: Allow" and add it to a new field called "Action".

This is what I have within my logstash config:

filter {
  if [service] == "network" {
    #mutate {
    # copy => { "properties.msg" => "Action:*"}
    #}
    mutate {
     split => {"properties.msg" => "Action:"}
    }
    mutate {
     add_field => { "Action" => "%{[properties.msg][1]}"}
    }
  }
}

With this config, I am getting this:

image

Help would be much appreciated. Thanks.

2

You don't have a field properties.msg. That would mean that there is a dot within the field name, not that there is a subfield. It's [properties][msg] :slight_smile:

3

Will this work for you?

filter {
  if [service] == "network" {
      grok {
           match => {
               "[properties][msg]" => "%{GREEDYDATA:preData}\s*Action\:\s*%{WORD:Action}"
           }
      }
  }
}
4

Hi Jenni,

Ahh my bad. So I have changed the logstash config to:

filter {
  if [service] == "network" {
    mutate {
     split => {"properties.msg" => "Action:"}
    }
    mutate {
     add_field => { "Action" => "%{[properties][msg][1]}"}
    }
  }
}

And now getting the output on Kibana as:

image

5

Hi Kin,

Interesting method. I will give this a go as well. Many thanks.

6

You'd have to change this at both places. And have a look at the dissect filter which would be a prettier solution. Grok is actually a bit much for such an easy split because it uses regular expressions where they are not neccessary.

7

@kelk unfortunately that config didnt produce anything

8

Here you go:

dissect {
 mapping => {
   "[properties][msg]" => "%{[properties][msg]} Action: %{Action}"
 }
}
2 Likes
9

@Jenni works like a charm! Much appreciated. Thanks.

10

sorry, I put
"message" as it is more common. Updated with "[properties][msg]". Hopefully it should work now

11

@kelk Thanks.

Another thing, if I wanted to do the same for the same field i.e.

"properties": {
      "msg": "UDP request from x.x.x.x:62151 to x.x.x.x:53."

To add UDP request from x.x.x.x:62151 to new field called "UDP Request From"
AND to add UDP request to x.x.x.x:53 to new field called "UDP Request to"

Would this be as simple as using the disect filter again?

That the field output would be:

UDP Request From          x.x.x.x:62151

UDP Request To               x.x.x.x:53

UDP Action:                       Allow
12

Because I come from a regex background, I might do all in one shot using grok

UDP request from 192.168.1.1:36877 to 192.168.2.1:53. Action: Allow

filter {
  if [service] == "network" {
      grok {
           match => {
               "[properties][msg]" => "UDP request from %{IP:source_ip}\:%{NUMBER:source_port} to %{IP:dest_ip}\:%{NUMBER:dest_port}\.\s*Action\:\s*%{WORD:Action}"
           }
      }
  }
}
1 Like
13

If you do it in an extra step:

dissect {
  mapping => { "[properties][msg]" => "UDP request from %{UPD Request from} to %{UPD Request to}"}
}
mutate {
  gsub => ["UPD Request to", "\.$", ""]
}

Otherwise:

dissect {
  mapping => { "[properties][msg]" => "UDP request from %{UPD Request from} to %{UPD Request to}. Action: %{UPD Action}"}
}
1 Like
14

Hi @Jenni and @kelk,

That both worked. Thank you very much!

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.