Using Key-value(KV) with multiple Value splits

Elastic Stack Logstash
1

I have a wineventlog-application log which has (ie) 'EventCode=33210 EventRecordID=12345' then changes to session_id:69,server_principal_id:226,etc etc so from = to : with , instead of spaces. Is there a way I can use the one Key-value processor to handle both?

2

Can you show a message not just few fields?

3

<15>Nov 02 14:44:04 AB23HJK120.example.co.uk WinEventLog:Application: EventCode=35203 EventRecordID=234567 Level=0 source=WinEventLog:Application EventID=23435 signature_id=23456 "audit_schema_version:1,event_time:2023-11-02 14:44:04.06423845,sequence_number:1,action_id:IN ,succeded:true,is_column_permission:false,session_id:49,server_principal_id:344,additional_information:<sql_stack>,frame nest_level = '1' ...

4

This is generally what a typical log will look like, I have shortened it as the middle part is more the same, but it also looks like poor truncation as each log ends with ... so I cant really dissect/grok the log using the first quotes with some end quotes. But you can get the gist of the poorly structured log here.

5

Did something that half works:
Within a Key-value(KV) processor:
Field split:
(\s(?=[{}a-zA-Z0-9]+=))|(,(?=[{}a-zA-Z0-9]+:))|((?<!:[{}a-zA-Z0-9]+)\s(?=[{}a-zA-Z0-9]+:))
Value split:
=|:

6

only thing was that it breaks when there was another event_time:'2023-11-02 14:44:04:06275475' in the log - it brings out 14 as a field with contents: 44:04:06275474
(for example)

7

Here is your code:

input {
  generator {
       message => '<15>Nov 02 14:44:04 AB23HJK120.example.co.uk WinEventLog:Application: EventCode=35203 EventRecordID=234567 Level=0 source=WinEventLog:Application EventID=23435 signature_id=23456 "audit_schema_version:1,event_time:2023-11-02 14:44:04.06423845,sequence_number:1,action_id:IN ,succeded:true,is_column_permission:false,session_id:49,server_principal_id:344,additional_information:<sql_stack>,frame nest_level = \'1\'"'
	   count => 1
  }
}

filter {

   dissect { mapping => { "message" => '<%{procid}>%{timestamp} %{+timestamp} %{+timestamp} %{host} %{wineventlog}:%{+wineventlog}: %{[@metadata][kvmsg]} "%{[@metadata][kvmsg2]}"' } }
   
   kv { 
      source => "[@metadata][kvmsg]"
      value_split => "="
      field_split => " "
   }

   kv { 
      source => "[@metadata][kvmsg2]"
      value_split => ":"
      field_split => ","
   }

      date {
        match => ["event_time", "yyyy-MM-dd HH:mm:ss.SSSSSSSS"]
		target=> "@timestamp" 
      }

   mutate {  remove_field => ["message", "event"] }
}

output {
    stdout { codec => rubydebug{ metadata => false}} # change to true to see metadata
}

Result:

{
                "@timestamp" => 2023-11-02T13:44:04.064Z,
       "server_principal_id" => "344",
                    "procid" => "15",
                    "source" => "WinEventLog:Application",
                 "action_id" => "IN ",
           "sequence_number" => "1",
      "is_column_permission" => "false",
                "session_id" => "49",
             "EventRecordID" => "234567",
                  "@version" => "1",
                      "host" => "AB23HJK120.example.co.uk",
               "wineventlog" => "WinEventLog:Application",
              "signature_id" => "23456",
                   "EventID" => "23435",
                 "EventCode" => "35203",
                "event_time" => "2023-11-02 14:44:04.06423845",
                 "timestamp" => "Nov 02 14:44:04",
                     "Level" => "0",
    "additional_information" => "sql_stack",
      "audit_schema_version" => "1",
                  "succeded" => "true"
}

You can use grok, however this is an internal log, no need much for validation such as IP, hostname,... so dissect would be good option.