Logstash mutate not working after kv filter applied

[splitmessage88]

Hi,
I created a filter for cisco syslog, input source cisco FTD.
All the fields are parsing correctly but I can't rename/remove fields with mutate after I used the kv {} filter.

1. Is it possible to use mutate after kv filter, or do I need to use grok patterns then mutate?
2. I'm able to remove the "data" field from ES output, is it possible to only remove fields, not change them?

filter {
  if [type] == "ling_ftd" {
    grok {
      match => { "message" => "(?<timestamp>%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}) %{WORD:hosthostname}%{SPACE}%(.*?):%{SPACE} %{GREEDYDATA:data}" }
      overwrite => [ "data" ]
  }
    kv {
      source => "data"
      field_split => ","
      value_split => ":"
 }
    mutate {
      rename => [
        "Client", "client",
        "DstIP", "destination.ip"
        ]
    remove_field => ['data']
 }
}
  else if [type] == "trend_dsm" {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
  }
 }
  else if [type] == "meraki" {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
  }
 }
}

Some sample output

  "_version": 1,
    "_score": 1,
    "_ignored": [
      "event.original.keyword",
      "message.keyword"
    ],
    "_source": {
      " EgressVRF": "test",
      " ResponderBytes": "46",
      " ApplicationProtocol": "ICMP",

"fields": {
      "EventPriority.keyword": [
        "Low"
      ],
      " IngressVRF": [
        "Global"
      ],
      " InstanceID.keyword": [
        "3"
      ],

.......
  ],
      " DstIP.keyword": [
        "10.x.x.x"

.....
 ],
      " ACPolicy": [
        "Policy"
      ],
      "EventPriority": [
        "Low"
      ],
      "timestamp": [
        "May 03 2022 15:03:09"
      ],
      " DstIP": [
        "10.x.x.x"

Thanks.

[leandrojmp (Leandro Pereira)]

You need to provide an example message so people can try to replicate your issue.

[splitmessage88]

Hello leandrojmp,

Sample message below

message
May 03 2022 15:03:09 hostname %FTD-1-4123: EventPriority: Low, DeviceUUID: 123-456-789-146-a34612, InstanceID: 3, FirstPacketSecond: 2022-05-03T15:03:09Z, ConnectionID: 536, AccessControlRuleAction: Allow, SrcIP: 10.10.10.2, DstIP: 192.168.52.71, ICMPType: Echo Request, ICMPCode: No Code, Protocol: icmp, IngressInterface: outside, EgressInterface: test

Thank you.

[leandrojmp (Leandro Pereira)]

All your field names extract from the data field, except the first one, have a leading space, your mutate is not working because the fields "Client" or "DstIp" don't exist, you have " Client" and " DstIp".

Try to change your field_split in the kv filter to ", "

Also, the rename option from the mutate filter expects a hash and the correct way to reference to nested fields in logstash is [top][nested], not top.nested, the second way will create a field with a literal dot in the name.

Your mutate should be.

mutate {
    rename => {
        "Client" => "client"
        "DstIp" => "[destination][ip]"
   }
    remove_field => ["data"]
}

[splitmessage88]

Thank you for the support leandrojmp!

I changed the field_split in the kv filter to ", " and used your mutate proposal and it solved my problem. I didn't see the space at all.

Is it possible to set a configuration to ignore spaces so this doesn't happen again?

[Badger]

@splitmessage88

Is it possible to set a configuration to ignore spaces so this doesn't happen again?

Another approach is to use the trim_key and trim_value options on the filter to remove these if they are not always present.

[splitmessage88]

Hi Badger,

Thank you for the advice!

Now the script works.

 kv {
      source => "data"
      trim_key => "\s"
      field_split => ","
      value_split => ":"
 }