Parsing KV filter is not working

Elastic Stack Logstash
1

Hi,

I need to parse each field from a McAfee WebGateway like having a field url, ip source, destination... Can you please help me to understand my mistakes with kv and grok? I do not success to make the kv works. Here is what I did:

  • Log exemple:

"<30>Apr 3 14:15:55 MWG05 mwg: [03/Apr/2019:14:15:55 +0200]#jean-bombeur#192.168.1.1#13.117.5.81#90#200#TCP_MISS_RELOAD#GET http://api.bing.com/qsml.aspx?query=troll&maxwidth=32765&rowheight=20&sectionHeight=160&FORM=IESS02&market=fr-FR HTTP/1.1# #Search Engines, Internet Services# #Minimal Risk# #text/xml#1036#1315#Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko# #-# #-# #0# #0# #-# #-# #-# #GRP-INTERN#"

  • part of the Logstash.conf 
      else if [host] in ["192.168.10.1", "192.168.10.2"] {
        # mcafee
        grok {
          match => ["message", "%{SYSLOG5424PRI}%{GREEDYDATA:data}"]
          add_tag => ["REMARQUABLE"]
        }
        kv {
            source => "data"
            field_split => "#"
        }
      }
  • Here is the actual result 
{
                                       "host" => "192.168.10.1",
                                       "data" => "Apr  3 14:15:55 MWG05 mwg: [03/Apr/2019:14:15:55 +0200]#jean-bombeur#192.168.1.1#13.117.5.81#90#200#TCP_MISS_RELOAD#GET http://api.bing.com/qsml.aspx?query=troll&maxwidth=32765&rowheight=20&sectionHeight=160&FORM=IESS02&market=fr-FR HTTP/1.1# #Search Engines, Internet Services# #Minimal Risk# #text/xml#1036#1315#Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko# #-# #-# #0# #0# #-# #-# #-# #GRP-INTERN#",
    "GET http://api.bing.com/qsml.aspx?query" => "troll&maxwidth=32765&rowheight=20&sectionHeight=160&FORM=IESS02&market=fr-FR HTTP/1.1",
                                    "message" => "<30>Apr  3 14:15:55 MWG05 mwg: [03/Apr/2019:14:15:55 +0200]#jean-bombeur#192.168.1.1#13.17.5.81#90#200#TCP_MISS_RELOAD#GET http://api.bing.com/qsml.aspx?query=troll&maxwidth=32765&rowheight=20&sectionHeight=160&FORM=IESS02&market=fr-FR HTTP/1.1# #Search Engines, Internet Services# #Minimal Risk# #text/xml#1036#1315#Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko# #-# #-# #0# #0# #-# #-# #-# #GRP-INTERN#",
                                       "type" => "mcafee",
                                   "@version" => "1",
                                 "@timestamp" => 2019-04-03T12:01:19.026Z,
                                       "tags" => [
        [0] "REMARQUABLE"
    ],
                             "syslog5424_pri" => "30"
}
2

data does not appear to be key/value data. What output are you expecting?

1 Like
3

Of course, I should have seen it. Sorry for this mistake and thank you for your time.
Since I can modify the log format from McAfee, I did this (it looks fine to me but if you have any idea to improve, feel free to share it):

Source log

<30>Apr 4 16:59:41 MGW02 mwg: #date|[04/Apr/2019:16:59:41 +0200]#account|jean-bombeur#srcip|192.168.1.1#dstip|172.64.238.38#timing_trans|44#statuscode|301#cache|TCP_MISS_RELOAD#request|HEAD Greenshot HTTP/1.1# #Shareware/Freeware#reputation|Minimal Risk# #-#331#559#-# #-#virus_name|-# #0#block_id|0# #-# #-# #-#rule_set|GRP-INTERN#

Part of the logstash.conf

  else if [host] in ["192.168.10.1", "192.168.10.2","192.168.20.1", "192.168.20.2",] {
    # mcafee
    grok {
      match => ["message", "%{SYSLOG5424PRI}%{GREEDYDATA:data}"]
    }
    kv {
        source => "data"
        field_split => "#"
        value_split => "|"
    }
    mutate {
      remove_field => ["data"]
    }
    if [host] in ["192.168.10.1", "192.168.10.2"] {
      mutate {
        add_field => { "country" => "Germany" }
      }
    }
    if [host] in ["192.168.20.1", "192.168.20.2"] {
      mutate {
        add_field => { "country" => "USA" }
      }
    }
  }

Actual result

{
             "srcip" => "192.168.1.1",
        "virus_name" => "-",
          "block_id" => "0",
           "message" => "<30>Apr  4 16:59:41 MGW02 mwg: #date|[04/Apr/2019:16:59:41 +0200]#account|jean-bombeur#srcip|192.168.1.1#dstip|172.64.238.38#timing_trans|44#statuscode|301#cache|TCP_MISS_RELOAD#request|HEAD http://getgreenshot.org/project-feed/ HTTP/1.1# #Shareware/Freeware#reputation|Minimal Risk# #-#331#559#-# #-#virus_name|-# #0#block_id|0# #-# #-# #-#rule_set|GRP-INTERN#",
              "date" => "04/Apr/2019:16:59:41 +0200",
             "cache" => "TCP_MISS_RELOAD",
              "host" => "192.168.10.1",
              "type" => "mcafee",
        "statuscode" => "301",
    "syslog5424_pri" => "30",
             "dstip" => "172.64.238.38",
           "request" => "HEAD http://getgreenshot.org/project-feed/ HTTP/1.1",
        "reputation" => "Minimal Risk",
      "timing_trans" => "44",
           "account" => "jean-bombeur",
          "rule_set" => "GRP-INTERN",
              "country" => "Germany",
        "@timestamp" => 2019-04-04T14:45:02.605Z,
          "@version" => "1"
}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.