Grok filter works in debugger but not in logstash

Elastic Stack Logstash
1

Hi,

I want to monitor windows registry event.
I have a grok filter to extract 2 values of registry path and put them in matching fields.

The target field is winlog.event_data.ObjectName

  • Kibana displayed value: \REGISTRY\MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{9459C573-B17A-45AE-9F64-1857B5D58CEE}

  • Json displayed value: \\REGISTRY\\MACHINE\\SOFTWARE\\Microsoft\\Active Setup\\Installed Components\\{9459C573-B17A-45AE-9F64-1857B5D58CEE}

filter {
  #we tells logstash to lookup for the csv if tags field contains windowsregistry string
  if [tags] =~ "windowsregistry" {
    grok { match => { "[winlog][event_data][ObjectName]" => "\\REGISTRY\%{WORD:[winlog][event_data][RegistryType]}\\%{WORD:[winlog][event_data][RegistryHiveType]}\\" } }
  }
}

I've tried with double \\ or simple \ or even \\\\ but none of them works. Do you have any ideas ?

Also Logstash in its error logs, read the escape \ as if its doubled amount.
If I double \\ logstash reads \\\\, If I \\\\ logstash reads \\\\\\\\

Note: I'm migrating from previous solution Graylogs where grok patterns used is this one (with \\\\ as escape:

image
image1738×77 8.04 KB

Thank you !

2

There are two changes you need to make to this

 match => { "[winlog][event_data][ObjectName]" => "\\REGISTRY\\%{WORD:[winlog][event_data][RegistryType]}\\%{WORD:[winlog][event_data][RegistryHiveType]}\\."

Note the second backslash after REGISTRY, and the . added at the end of the pattern. You cannot escape a backslash at the end of a quoted string. The configuration parser will always interpret that as escaping the quote. So ...

input { generator { count => 1 lines => [ '\REGISTRY\MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{9459C573-B17A-45AE-9F64-1857B5D58CEE}' ] } }

output { stdout { codec => rubydebug { metadata => false } } }
filter { 
    mutate { rename => { "message" => "[winlog][event_data][ObjectName]" } }
    grok { match => { "[winlog][event_data][ObjectName]" => "\\REGISTRY\\%{WORD:[winlog][event_data][RegistryType]}\\%{WORD:[winlog][event_data][RegistryHiveType]}\\." } }
}

will produce

    "winlog" => {
    "event_data" => {
            "RegistryType" => "MACHINE",
        "RegistryHiveType" => "SOFTWARE",
              "ObjectName" => "\\REGISTRY\\MACHINE\\SOFTWARE\\Microsoft\\Active Setup\\Installed Components\\{9459C573-B17A-45AE-9F64-1857B5D58CEE}"
    }
},
3

Thank you it work BUT I had to remove the if statement do make it work, I searching why it is not working with my condition.

I would prefer to run the filter only if this tags appears, otherwise it will try to apply the filter on every event log.

filter {
  #we tells logstash to lookup for the csv if tags field contains windowsregistry string
#  if [tags] =~ "windowsregistry" {

    grok { match => { "[winlog][event_data][ObjectName]" => "\\REGISTRY\\%{WORD:[winlog][event_data][RegistryType]}\\%{WORD:[winlog][event_data][RegistryHiveType]}\\." } }


    translate {
      source => "[winlog][event_data][OperationType]"
      target => "[winlog][event_data][OperationTypeDescription]"
      dictionary_path => "/usr/share/logstash/pipeline/winlogbeat/csv/Windows-Registry-Code-to-CodeDescription.csv"
      refresh_interval => 60
      refresh_behaviour => replace
      fallback => "No Code Description Found"
     }


    translate {
      source => "[winlog][event_data][OldValueType]"
      target => "[winlog][event_data][OldValueTypeDescription]"
      dictionary_path => "/usr/share/logstash/pipeline/winlogbeat/csv/Windows-Registry-Code-to-CodeDescription.csv"
      refresh_interval => 60
      refresh_behaviour => replace
      fallback => "No Code Description Found"
     }


    translate {
      source => "[winlog][event_data][NewValueType]"
      target => "[winlog][event_data][NewValueTypeDescription]"
      dictionary_path => "/usr/share/logstash/pipeline/winlogbeat/csv/Windows-Registry-Code-to-CodeDescription.csv"
      refresh_interval => 60
      refresh_behaviour => replace
      fallback => "No Code Description Found"
     }

  #}
}
4

[tags] is an array. If you want to test whether one of the array entries is "windowsregistry" then use

if "windowsregistry" in [tags] {...
1 Like
5

I was reading this info on the doc because I tested with if [event][action] =~ "Registry" it works.

And because one of my other filters if condition contains also a regex match like this:
if [winlog][channel] =~ "Security" {

And after I realized that its an array field and not [winlog][channel]

Some details are tricky ! I just need time to get used to it.

Thank you again !