Logstash KV filter - Wrong parsing

[vasek (Václav Šulc)]

Hello,
I'm using KV filter in Logstash. Logstash configuration:

filter {


  #PARSING LOG
  #------------------------------------------

  grok {
      patterns_dir => ["/etc/logstash/grok-patterns"]
      break_on_match => true
      match => { "message" => [ "%{CYBERARC_LOG}" ] }
      overwrite => [ "message" ]
      tag_on_failure => ["not_parsed", "not_parsed_cyberarc"]
  }


  #MAPPING DATE
  #------------------------------------------

  if [timestamp] {

    #https://discuss.elastic.co/t/logstash-date-parse-failure-for-jdbc-input/92713/2
    #https://github.com/logstash-plugins/logstash-filter-date/issues/95 

    mutate {
      convert => { "timestamp" => "string" }
    }

    date {
      locale => "en"
      match => ["timestamp", "ISO8601", "yyyy-MM-dd'T'HH:mm:ss" ]
      target => "@timestamp"
      remove_field => ["timestamp"]
    }
  }

  # KV FILTER
  #------------------------------------------


  if [cef_message] {
    kv {
      source => "cef_message"
      transform_key => "lowercase"
      trim_key => "no"
      trim_value => "no"
      field_split => " "
      whitespace => "strict"
      prefix => "kv_"
      #remove_field => ["cef_message"]
    }
  }
}

There is a log:

<5>1 2021-08-24T09:13:26Z CDIS000PHANT642 CEF:0|Cyber-Ark|Vault|12.2.0000|99|Open File|5|act=\"Open File\" fname=\"Root\\Policies\\Policy-CyberArkPTA.ini\" cs1Label=\"Affected User Name\" cs1=\"test\" cs2Label=\"Safe Name\" cs2=\"PasswordManagerShared\" cs3Label=\"Device Type\" cs3=\"test\" cs4Label=\"Other information message\" cs4=\"information\" cs5Label=\"Other information\" cs5=\"test\" cn1Label=\"Request Id\" cn1=\"test\" cn2Label=\"Ticket Id\" cn2=\"test\" 

From message was parsed a field with name cef_message.

"cef_message": "act=\"Open File\" fname=\"Root\\Policies\\Policy-CyberArkPTA.ini\" cs1Label=\"Affected User Name\" cs1=\"test\" cs2Label=\"Safe Name\" cs2=\"PasswordManagerShared\" cs3Label=\"Device Type\" cs3=\"test\" cs4Label=\"Other information message\" cs4=\"information\" cs5Label=\"Other information\" cs5=\"test\" cn1Label=\"Request Id\" cn1=\"test\" cn2Label=\"Ticket Id\" cn2=\"test\" "

KV filter was applied on cef_field field.

Very strange behaving is that information is trimmed to informati.

    "kv_act": "Open File"
    "kv_cn1": "test",
    "kv_cn1label": "Request Id",
    "kv_cn2": "test",
    "kv_cn2label": "Ticket Id",
    "kv_cs1": "test",
    "kv_cs1label": "Affected User Name",
    "kv_cs2": "PasswordManagerShared",
    "kv_cs2label": "Safe Name",
    "kv_cs3": "test",
    "kv_cs3label": "Device Type",
    "kv_cs4": "informati",
    "kv_cs4label": "Other information message",
    "kv_cs5": "test",
    "kv_cs5label": "Other informati",
    "kv_fname": "Root\\Policies\\Policy-CyberArkPTA.ini",

Look at fields:

  "kv_cs5label": "Other informati"
  "kv_cs4label": "Other information message"
  "kv_cs4": "informati"

If the text ends with "on", it removes the text "on".

I'm using Logstash 7.8.1. Can anyone help me to resolve this problem?

[Cad]

Hi,

I think the error is because of the trim_value => "no". Indeed, the code for the trim with your values give a regex like this ^[no]+|[no]+$ and this regex on the word information match the two last values.
So the trim_value is a string who contains character you want to remove, it don't only remove the string itself.

[leandrojmp (Leandro Pereira)]

Also, your input seems to be a syslog message with a CEF part, you can try the cef codec directly in your input, this way you won't need neither grok nor kv, the cef codec would parse your message.

[vasek (Václav Šulc)]

Hi @Cad,
thank you for your solution! I thought that trim_value was supposed to take the value of a boolean - aaa, such as mistake .

[vasek (Václav Šulc)]

Hi @leandrojmp . Thank you for the tip I'll try it.