# Not sure if a field is nill, null or empty. Help please **URL:** \ **Category:** Logstash\ **Created:** [April 23, 2021, 8:04pm UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123 "2021-04-23T20:04:50Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Post date:** [April 23, 2021, 8:04pm UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123/1 "2021-04-23T20:04:50Z")
Hi All, It's about barracuda FW log with the following entry `+02:00 Security XXX Block: type=FWD|proto=TCP|srcIF=eth0|srcIP=0.0.0.0|srcPort=58570|srcMAC=00:00:00:00:00:00|dstIP=0.0.0.0|dstPort=3002|dstService=exlm-agent|dstIF=|rule=L2GIS:|info=Block no Rule Match|srcNAT=0.0.0.0|dstNAT=0.0.0.0|duration=0|count=1|receivedBytes=0|sentBytes=0|receivedPackets=0|sentPackets=0|user=|protocol=|application=|target=|content=|urlcat=` As you can see **urlcat** (the last field) end with nil, null or empty, not exactly sure. My **rubydebug** output filter returns ![7](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cbb3c25e5735bd56729bfe8f22fc5c506456974a.png) The part of the regex I use for parsing is: `urlcat=(?[\w\s\d()]+|)` I tried with **mutate** and **ruby** to substitute the empty field with NA but it wont work. I tried ``` ruby { code => "if event.get('URLCategory').nil?; event.set('tags', 'null-value');end" } if "null-value" in [tags] { mutate { update => { "URLCategory" => "NA" } } } ``` `ruby { code => "event['URLCategory'] = 'empty' if event['URLCategory'] == nil;" }` `mutate { gsub => ["URLCategory", " ", "NA"] }` ``` ruby { code => ' event.to.hash.each { |k, v| if v == '' # Also "" event.set(k, "NA") end } ' } ruby { code => "if event.get('location').nil?; event.set('tags','null-value');end" } if "null-value" in [tags] { do something } ruby { code => ' event.to.hash.each { |k, v| if v.nil? # Also v.empty? and v == nil, "" event.set(k, "NA") end } ' } ``` Funny thing is that every other field that ends with | i did with ruby and it works just fine substituting | with N/A ``` ruby { code => ' event.to_hash.each { |k, v| if v == "|" event.set(k, "N/A") end } ' } ``` Could you please help. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 23, 2021, 8:40pm UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123/2 "2021-04-23T20:40:51Z")
What does the UrlCategory field look like in the rubydebug output? Note that your regexp has UrlCategory but all your filters use URLCategory, which does not match. Instead of ``` mutate { gsub => ["URLCategory", " ", "NA"] } ``` I suspect you want ``` mutate { gsub => ["UrlCategory", "", "NA"] } ``` ---
**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Post date:** [April 24, 2021, 5:26am UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123/3 "2021-04-24T05:26:18Z")
Thank you @Badger My **rubydebug** output @message returns `"message" => "+02:00 Security XXX Block: type=FWD|proto=TCP|srcIF=eth0|srcIP=0.0.0.0|srcPort=58570|srcMAC=00:0d:48:51:33:24|dstIP=0.0.0.0|dstPort=3002|dstService=exlm-agent|dstIF=|rule=L2GIS:|info=Block no Rule Match|srcNAT=0.0.0.0|dstNAT=0.0.0.0|duration=0|count=1|receivedBytes=0|sentBytes=0|receivedPackets=0|sentPackets=0|user=|protocol=|application=|target=|content=|urlcat="` Otherwise I tried your solutions and it won't work. If `urlcat=` it did nothing if urlcat is not empty, e.g. `urlcat=Categorization in Progress (98)"` It messed it up as it shown Below is my rubydebug output (partial, for urlcat only) `"URLCategory" => "NACNAaNAtNAeNAgNAoNArNAiNAzNAaNAtNAiNAoNAnNA NAiNAnNA NAPNArNAoNAgNArNAeNAsNAsNA NA(NA9NA8NA)NA",` I'll paste a two FW entry - one without urlcat value and one with `02:00 Security XXX Block: type=FWD|proto=TCP|srcIF=eth0|srcIP=0.0.0.0|srcPort=58570|srcMAC=00:0d:48:51:33:24|dstIP=0.0.0.0|dstPort=3002|dstService=exlm-agent|dstIF=|rule=L2GIS:|info=Block no Rule Match|srcNAT=0.0.0.0|dstNAT=0.0.0.0|duration=0|count=1|receivedBytes=0|sentBytes=0|receivedPackets=0|sentPackets=0|user=|protocol=|application=|target=|content=|urlcat=` `+02:00 Security daeifw01 Block: type=FWD|proto=TCP|srcIF=eth0|srcIP=10.21.88.146|srcPort=58570|srcMAC=00:0d:48:51:33:24|dstIP=10.22.136.128|dstPort=3002|dstService=exlm-agent|dstIF=|rule=L2GIS:|info=Block no Rule Match|srcNAT=0.0.0.0|dstNAT=0.0.0.0|duration=0|count=1|receivedBytes=0|sentBytes=0|receivedPackets=0|sentPackets=0|user=|protocol=|application=|target=|content=|urlcat=Categorization in Progress (98)` I also paste my conf file filter plugin ``` filter { if [message] =~ "Security" or [message] =~ "Info" { grok { match => { "message" => "(?:Z|[+-]%{HOUR}(?::?%{MINUTE})) (?\w+) (?[a-zA-Z0-9]+) (?\w+): type=(?\w+|\|)\|?proto=(?\w+|\|)\|?srcIF=(?[\w\s\d\.]+|\|)\|?srcIP=(?[\d\.]+|\|)\|?srcPort=(?[\d\.]+|\|)\|?srcMAC=(?[\d:\w]+|\|)\|?dstIP=(?[\d\.]+|\|)\|?dstPort=(?[\d\.]+|\|)\|?dstService=(?[\w\d\.-]+|\|)\|?dstIF=(?[\w\d\.]+|\|)\|?rule=(?[\w\s=:<>-]+|\|)\|?info=(?[\w\d\s]+|\|)\|?srcNAT=(?[\d\.]+|\|)\|?dstNAT=(?[\d\.]+|\|)\|?duration=(?[\d]+|\|)\|?count=(?[\d]+|\|)\|?receivedBytes=(?[\d]+|\|)\|?sentBytes=(?[\d]+|\|)\|?receivedPackets=(?[\d]+|\|)\|?sentPackets=(?[\d]+|\|)\|?user=(?[\w]+|\|)\|?protocol=(?[\w]+|\|)\|?application=(?[\w\s]+|\|)\|?target=(?[\w\s\.]+|\|)\|?content=(?[\w\s]+|\|)\|?urlcat=(?[\w\s\d()]+|)" } } #mutate { update => { "URLCategory" => "NA" } } #mutate { gsub => ["URLCategory", "", "NA"] } #ruby { code => "if event.get('URLCategory').nil?; event.set('tags', 'null-value');end" } #if "null-value" in [tags] { mutate { update => { "URLCategory", => "NA" } } } #ruby { # code => "event['URLCategory'] = 'empty' if event['URLCategory'] == nil;" #} #ruby { # code => ' # event.to.hash.each { |k, v| # if v == nil # event.set(k, "NA") # end # } # ' # } ruby { code => ' event.to_hash.each { |k, v| if v == "|" event.set(k, "N/A") end } ' } #ruby { code => "if event.get('URLCategory').nil?; event.set('tags', 'null-value');end" } #if "null-value" in [tags] { mutate { update => { "URLCategory" => "NA" } } } #if "null-value" in [tags] { mutate { coerce => { "URLCategory" => "Nnnnn" } } } #if "null-value" in [tags] { mutate { gsub => ["URLCategory", "", "nnnnnnn"] } } #if [URLCategory] == "" { #mutate { update => { "URLCategory" => "NA" } } #} cidr { add_tag => ["SomeTAG"] address => ["%{SourceIP}"] network => ["10.0.0.0/8"] } if "someTAG" not in [tags] { drop { } } if [L7Protocol] == "DNS" or [L7Protocol] == "SMB" { prune { whitelist_names => ["^L7Protocol$", "^Application$", "^URLCategory$"] } } else if [Action] == "Block" { prune { blacklist_names => ["^Action$"] } } else { drop { } } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 24, 2021, 5:27pm UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123/4 "2021-04-24T17:27:28Z")
> [@stillfreem](#): > > "URLCategory" =\> "NACNAaNAtNAeNAgNAoNArNAiNAzNAaNAtNAiNAoNAnNA NAiNAnNA NAPNArNAoNAgNArNAeNAsNAsNA NA(NA9NA8NA)NA" In this case the gsub has inserted "NA" between every pair of letters in the message "Categorization in Progress (98)". If you do not set the grok option "keep\_empty\_captures =\> true" then if the capture group is "" then no field is added to the message. In that case you could use ``` if ! [URLCategory] { mutate { add_field => { "URLCategory" => "NA" } } ``` If you set that option then [URLCategory] will be "" and you could use ``` mutate { gsub => ["URLCategory", "^$", "NA"] } ``` Lastly, instead of using a grouping with + (one or more) and an alternation with the empty pattern `(?[\w\s\d()]+|)` I would use \* (zero or more) `(?[\w\s\d()]*)`. Both work. ---
**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Post date:** [April 24, 2021, 6:55pm UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123/5 "2021-04-24T18:55:29Z")
You're awesome @Badger, It worked as a charm, seems that I need to review the documentation and not just flick through it 🙂 Regarding my regex, I'll try to implement wherever possible the **zero or more** option instead of grouping + **one or more** and alternation with an empty pattern. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 22, 2021, 6:55pm UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123/6 "2021-05-22T18:55:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.