Not sure if a field is nill, null or empty. Help please

stillfreem · April 23, 2021, 8:04pm

As you can see urlcat (the last field) end with nil, null or empty, not exactly sure.

My rubydebug output filter returns

The part of the regex I use for parsing is:
urlcat=(?<UrlCategory>[\w\s\d()]+|)

I tried with mutate and ruby to substitute the empty field with NA but it wont work.
I tried

ruby { code => "if event.get('URLCategory').nil?; event.set('tags', 'null-value');end" }
if "null-value" in [tags] { mutate { update => { "URLCategory" => "NA" } } }

ruby { code => "event['URLCategory'] = 'empty' if event['URLCategory'] == nil;" }

mutate { gsub => ["URLCategory", " ", "NA"] }

ruby {
                      code => '
                        event.to.hash.each { |k, v|
                          if v == '' # Also ""
                             event.set(k, "NA")
                          end
                        }
                              '
                      }

  ruby {
    code => "if event.get('location').nil?; event.set('tags','null-value');end"
}
if "null-value" in [tags] { do something }

ruby {
                      code => '
                        event.to.hash.each { |k, v|
                          if v.nil? # Also v.empty? and v == nil, ""
                             event.set(k, "NA")
                          end
                        }
                              '
                      }

Funny thing is that every other field that ends with | i did with ruby and it works just fine substituting | with N/A

ruby {
                      code => '
                        event.to_hash.each { |k, v|
                           if v == "|"
                              event.set(k, "N/A")
                           end
                        }
                              '
                      }

Could you please help.

Badger · April 23, 2021, 8:40pm

What does the UrlCategory field look like in the rubydebug output?

Note that your regexp has UrlCategory but all your filters use URLCategory, which does not match.

Instead of

mutate { gsub => ["URLCategory", " ", "NA"] }

I suspect you want

mutate { gsub => ["UrlCategory", "", "NA"] }

stillfreem · April 24, 2021, 5:26am

Thank you @Badger

Otherwise I tried your solutions and it won't work.
If urlcat= it did nothing
if urlcat is not empty, e.g.
urlcat=Categorization in Progress (98)"
It messed it up as it shown Below is my rubydebug output (partial, for urlcat only)
"URLCategory" => "NACNAaNAtNAeNAgNAoNArNAiNAzNAaNAtNAiNAoNAnNA NAiNAnNA NAPNArNAoNAgNArNAeNAsNAsNA NA(NA9NA8NA)NA",

I'll paste a two FW entry - one without urlcat value and one with

I also paste my conf file filter plugin

filter {
        if [message] =~ "Security" or [message] =~ "Info" {
                grok { match => { "message" => "(?:Z|[+-]%{HOUR}(?::?%{MINUTE})) (?<LogLevel>\w+) (?<SourceSystem>[a-zA-Z0-9]+) (?<Action>\w+): type=(?<Type>\w+|\|)\|?proto=(?<L4Protocol>\w+|\|)\|?srcIF=(?<SrcInterface>[\w\s\d\.]+|\|)\|?srcIP=(?<SourceIP>[\d\.]+|\|)\|?srcPort=(?<SourcePort>[\d\.]+|\|)\|?srcMAC=(?<SourceMAC>[\d:\w]+|\|)\|?dstIP=(?<DestinationIP>[\d\.]+|\|)\|?dstPort=(?<DestinationPort>[\d\.]+|\|)\|?dstService=(?<dstService>[\w\d\.-]+|\|)\|?dstIF=(?<dstIF>[\w\d\.]+|\|)\|?rule=(?<Rule>[\w\s=:<>-]+|\|)\|?info=(?<Info>[\w\d\s]+|\|)\|?srcNAT=(?<srcNAT>[\d\.]+|\|)\|?dstNAT=(?<dstNAT>[\d\.]+|\|)\|?duration=(?<Duration>[\d]+|\|)\|?count=(?<Count>[\d]+|\|)\|?receivedBytes=(?<ReceivedBytes>[\d]+|\|)\|?sentBytes=(?<SentBytes>[\d]+|\|)\|?receivedPackets=(?<ReceivedPackets>[\d]+|\|)\|?sentPackets=(?<SentPackets>[\d]+|\|)\|?user=(?<user>[\w]+|\|)\|?protocol=(?<L7Protocol>[\w]+|\|)\|?application=(?<Application>[\w\s]+|\|)\|?target=(?<Target>[\w\s\.]+|\|)\|?content=(?<Content>[\w\s]+|\|)\|?urlcat=(?<URLCategory>[\w\s\d()]+|)" } }

                #mutate { update => { "URLCategory" => "NA" } }

                #mutate { gsub => ["URLCategory", "", "NA"] }

                #ruby { code => "if event.get('URLCategory').nil?; event.set('tags', 'null-value');end" }
                #if "null-value" in [tags] { mutate { update => { "URLCategory", => "NA" } } }

                #ruby {
                #    code => "event['URLCategory'] = 'empty' if event['URLCategory'] == nil;"
                #}

                #ruby {
                #     code => '
                #       event.to.hash.each { |k, v|
                #         if v == nil
                #            event.set(k, "NA")
                #         end
                #       }
                #             '
                #      }
                ruby {
                      code => '
                        event.to_hash.each { |k, v|
                           if v == "|"
                              event.set(k, "N/A")
                           end
                        }
                              '
                      }

                #ruby { code => "if event.get('URLCategory').nil?; event.set('tags', 'null-value');end" }
                #if "null-value" in [tags] { mutate { update => { "URLCategory" => "NA" } } }
                #if "null-value" in [tags] { mutate { coerce => { "URLCategory" => "Nnnnn" } } }
                #if "null-value" in [tags] { mutate { gsub => ["URLCategory", "", "nnnnnnn"] } }

                #if [URLCategory] == "" {
                #mutate { update => { "URLCategory" => "NA" } }
                #}

                cidr {
                      add_tag => [ "SomeTAG" ]
                      address => [ "%{SourceIP}" ]
                      network => [ "10.0.0.0/8" ]
                     }

                if "someTAG" not in [tags] { drop { } }

                if [L7Protocol] == "DNS" or [L7Protocol] == "SMB" { prune { whitelist_names => ["^L7Protocol$", "^Application$", "^URLCategory$"] } }
                else if [Action] == "Block" { prune { blacklist_names => ["^Action$"] } }
                else { drop { } }
        }
}

Badger · April 24, 2021, 5:27pm

In this case the gsub has inserted "NA" between every pair of letters in the message "Categorization in Progress (98)".

If you do not set the grok option "keep_empty_captures => true" then if the capture group is "" then no field is added to the message. In that case you could use

if ! [URLCategory] { mutate { add_field => { "URLCategory" => "NA" } }

If you set that option then [URLCategory] will be "" and you could use

mutate { gsub => [ "URLCategory", "^$", "NA" ] }

Lastly, instead of using a grouping with + (one or more) and an alternation with the empty pattern (?<URLCategory>[\w\s\d()]+|) I would use * (zero or more) (?<URLCategory>[\w\s\d()]*). Both work.

stillfreem · April 24, 2021, 6:55pm

You're awesome @Badger, It worked as a charm, seems that I need to review the documentation and not just flick through it

Regarding my regex, I'll try to implement wherever possible the zero or more option instead of grouping + one or more and alternation with an empty pattern.

system · May 22, 2021, 6:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.