Issue with KV Filter when value ends with escape character

Elastic Stack Logstash
1

I am having an issue parsing some logs, and I would appreciate if someone could assist.

I have logs coming in that are primarily handled with a KV Filter. This works fine on about 99% of the messages, but one message type is giving me trouble. Here is a snippet from a sample log message giving me trouble:

File Name: foo.exe, Path: C:\, Drive Type: Internal Hard Drive

kv {
  transform_key => "lowercase"
  trim_key => " "   # Trim any leading or trailing spaces
  value_split => ":"
  field_split => "\,"
}

# Substitutes spaces in the key names for underscores
ruby {
  code => "
    event.to_hash.each {
      |key, value|
      if key =~ / / then
        event.set(key.gsub(' ','_'), event.remove(key))
      end
    }
  "
}

Since the "Path" contains my split character ':' in the value, it is not parsing properly. How can I handle a value that contains the split character? Using the above example, the message would come through like this:

file_name: foo.exe
     path: D:\, Drive Type: Internal Hard Drive

When I want it to come through like this:

 file_name: foo.exe
      path: D:\
drive_type: Internal Hard Drive

I would appreciate any thoughts on this.

2

The problem is not that the value contains the value_split character, the problem is that the message contains the field_split character escaped. It would help other folks if you adjusted the thread title.

The backslash in the message is escaping the comma so that it does not get matched by field_split. You could try using a mutate+gsub to change it into one of

Path: C:\\,
Path: C:\ ,
1 Like
3

@Badger

That was a poor example on my part. Let's say the path is 'C:\Program Files'. The backslash is not escaping any special characters in this instance. The issue is still the colon : Value Split character, not the comma.

4

With this configuration

input { generator { count => 1 lines => [ 'File Name: foo.exe, Path: C:\Program Files, Drive Type: Internal Hard Drive' ] } }
filter {
    kv {
        transform_key => "lowercase"
        trim_key => " "   # Trim any leading or trailing spaces
        value_split => ":"
        field_split => "\,"
    }
}
output { stdout { codec => rubydebug { metadata => false } } }

I get

 "file name" => "foo.exe",
      "path" => "C:\\Program Files",
"drive type" => "Internal Hard Drive",

What issue do you have with that?

5

Since I could not get this figured out before, I have actually already moved away from the KV filter in favor of Grok patterns on this one. There are only five different patterns for this source, so it was quick and easy to do.

I just tested again, and you are correct. D:\ and D:\SomePath\ both cause the issue while D:\SomePath does not. The comma is being escaped as you mentioned.

I appreciate your responses on this one, and if I run into the issue again, I will use your gsub suggestion to ensure the field_split character does not get escaped.

EDIT: Updated the post title.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.