Space in key : remove_char_key is not working

Elastic Stack Logstash
1

Hello !

I have a nested part in a log I can't succeed to parse :

Here is my config :

filter {

#replace second "msg" field by "nested" field
  mutate {
    gsub => ["message", "Message=", " nested="]
  }


  kv {
    source => "message"
    field_split => "    "
    value_split => "="
  }



  kv {
source => "nested"
remove_char_key => " "
value_split => ":"

  }
}

Result :

    {
        "<13>Mar 31 14:36:06 192.168.200.34 AgentDevice" => "WindowsLog",
                                              "Computer" => "WINCOLLECT_LOGSTASH",
                                          "RecordNumber" => "1467",
                                                 "Level" => "Log Always",
                                            "@timestamp" => 2020-03-31T12:36:14.998Z,
                                         "TimeGenerated" => "1585658165",
                                         "PluginVersion" => "7.2.9.96",
                                           "TimeWritten" => "1585658165",
                                                  "host" => "192.168.200.34",
                                              "Keywords" => "Audit Success",
                                               "message" => "<13>Mar 31 14:36:06 192.168.200.34 AgentDevice=WindowsLog\tAgentLogFile=Security\tPluginVersion=7.2.9.96\tSource=Microsoft-Windows-Security-Auditing\tComputer=WINCOLLECT_LOGSTASH\tOriginatingComputer=WINCOLLECT_LOGSTASH\tUser=\tDomain=\tEventID=4634\tEventIDCode=4634\tEventType=8\tEventCategory=12545\tRecordNumber=1467\tTimeGenerated=1585658165\tTimeWritten=1585658165\tLevel=Log Always\tKeywords=Audit Success\tTask=SE_ADT_LOGON_LOGOFF\tOpcode=Info\tnested=An account was logged off.  Subject:  Security ID:  WINCOLLECT_LOGS\\Administrator  Account Name:  Administrator  Account Domain:  WINCOLLECT_LOGS  Logon ID:  0xADD113  Logon Type:   7  This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.",
                                          "AgentLogFile" => "Security",
                                               "EventID" => "4634",
                                                  "port" => 50120,
                                             "EventType" => "8",
                                               "Subject" => "Security",
                                                  "Type" => "7",
                                                "Domain" => "WINCOLLECT_LOGS",
                                   "OriginatingComputer" => "WINCOLLECT_LOGSTASH",
                                         "EventCategory" => "12545",
                                           "EventIDCode" => "4634",
                                              "@version" => "1",
                                                    "ID" => [
            [0] "WINCOLLECT_LOGS\\Administrator",
            [1] "0xADD113"
        ],
                                                  "Name" => "Administrator",
                                                "nested" => "An account was logged off.  Subject:  Security ID:  WINCOLLECT_LOGS\\Administrator  Account Name:  Administrator  Account Domain:  WINCOLLECT_LOGS  Logon ID:  0xADD113  Logon Type:   7  This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.",
                                                  "Task" => "SE_ADT_LOGON_LOGOFF",
                                                  "type" => "syslog",
                                                "Opcode" => "Info",
                                                "Source" => "Microsoft-Windows-Security-Auditing"
    }

It shoud be "SecurityID" and not "ID"
It should be "AccountDomain" and not "Domain"
Ir should be "LogonType" and not "Type"

Thanks for your help !

2

You have the field_split character inside the keys. That will result in exactly what you see -- part of the key has no associated value and is therefore discarded.

3

There is no field_split setting for nested part (which the one I have issue with) :

  kv {
source => "nested"
remove_char_key => " "
value_split => ":"
  }

I taught that remove_char_key could remove the space before the key (ID in this example) to transform

"Security ID: WINCOLLECT_LOGS\Administrator" to "SecurityID: WINCOLLECT_LOGS\Administrator"

But it doens't work. Or maybe I didn't understood the meaning of remove_char_key ?

Do you know a way to figure this out ?

4

No, kv does not work the way you want it to work.

5

Mmmk. Ok.

So I'm asking how users who have the same issue (space in the key) can deal with it ?

6

@Christian_Dahlqvist I've read some similar post from you but this does'nt apply to my needs. Can you please just confirm what I want to do is not possible using kv filter ?

7

I agree with @Badger here.

8

Ok thanks. Is there exists any workaround ?

How could I parse this :

"nested" => "An account was logged off. Subject: Security ID: WINCOLLECT_LOGS\Administrator Account Name: Administrator Account Domain: WINCOLLECT_LOGS Logon ID: 0xADD113 Logon Type: 7 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.",

In this :

Security ID: WINCOLLECT_LOGS\Administrator
Account Name: Administrator
Account Domain: WINCOLLECT_LOGS
[...]

9

I would recommend either reformatting g tv bd message ahead of using the JVM filter or creating custom parsing through a ruby filter.

10

OK. I will give a try on ruby filter. Thank you @Christian_Dahlqvist and @Badger for your support :wink:

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.