Any filter for parsing the URL request from firewalls?

Elastic Stack Logstash
1

regarding payload within firewall logs, the request field has a single line separated by \r\n which I've made it into individual line. Any known parsers/filter to make this into key-value pairs?
(especially the User-Agent , host etc all into individual fields?) and then the cookie into individual fields as well

Please note below example, I've split lines to make it readable, but it is all into one-line with \r\n as field splitter.

Host: www.something.co.uk
Connection: keep-alive
User-Agent: Mozilla/5.0 (Linux; Android 8.1.0; DUB-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36
Accept: image/webp,image/apng,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://www.something.co.uk/assets/styles.css?20201008.2
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: check=true; AMCVS_BB2A12535131457C0A490D45%40AdobeOrg=1; AMCV_BB2A12535131457C0A490D45%40AdobeOrg=-1712354808%7CMCIDTS%7C18544%7CMCMID%7C01238256657182159320825076497090008339%7CMCAAMLH-1602767893%7C6%7CMCAAMB-1602767893%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1602170294s%7CNONE%7CMCAID%7CNONE%7CvVersion%7C4.3.0; mboxEdgeCluster=37; buyapowa_voucher_code=undefined; channeloriginator=seo; channelcloser=seo; VF-BINS=c7b152e2-4c0f-49aa-abe3-e657dc85e224; VF-SID=2748be33-f576-4248-a8b9-57639b73a394; VF-OPT=true; VF-UUID=42f94982-936e-4be1-a498-80a11195e9c2; _uetsid=cc30e5d0096811eb85c1e5508f491fe9; _uetvid=cc3342d0096811eba4063f93798707ac; PlatformAuthToken=PiwBWOulJRn%2FxEjZszyVdTPxfwioN5VzlOYYBTZMXj74iUXWOibanjacplhKwSNciIbBv2WMG8uANwiktrJe%2BLAhWkrm1SUkx2XfkctsBzPG7mqrkperzAbTdjnn%2FFg1X7Cms4NFLt1wpdFl6RYXmyOL%2BcXuP7XRE4cNrvGDKUTbiaOO7kK1z%2BuGro1qTkySjVbKq3hUsLBnkHq86cBd%2F3juJSvMibY92jqs5c3yb6rSDvld8Ysy0N7hJr9WDQ4wGa3LyjTeZIf8ng1TvH446tLHJVEmZBb0t%2FdM7sD0CUHsbrm%2FKa%2FS27LPCg9Ur4mb4hbmi8G3XOsVkeyYM2hEegeP%2B8a88eaa9da11daff849648423eb5d80c32eeed61570069d77a4edfd1; TS01056ccb030=01f849ee050c7e8614acb5e5278b840d830f072001bd7c6e633ebe76e6eebaf9ab435ceeae04fd395a3b1076a368d2cc7594b2e36e; _pin_unauth=dWlkPU1EaGlNelpsWVRRdFpHRm1OaTAwTkdSbUxXRmtZamN0TldVMFl6QXlObU00TVRsaA; AuthenticatedTracker=1602166729; Session=j%3A%7B%22platformSessionId%22%3A%220b2553b7-e9c1-4b3d-893b-1bcb63be21ea%22%2C%22subscriptionIdHash%22%3A%22f9a5a55a396eececfb97b49b%22%2C%22subscriptionType%22%3A%22something%22%2C%22assuranceLevel%22%3A2%2C%22givenName%22%3A%22SULTANA%20MUNNI%22%2C%22numberOfAccounts%22%3A1%2C%22numberOfSubscriptions%22%3A1%2C%22accountCategory%22%3A%22Individual%22%2C%22accountSubCategory%22%3A%22Consumer%22%7D; PlatformAccessToken=PiwBWOulJRn%2FxEjZszyVdTPxfwioN5VzlOYYBTZMXj74iUXWOibanjacplhKwSNciIbBv2WMG8uANwiktrJe%2BLAhWkrm1SUkx2XfkctsBzPG7mqrkperzAbTdjnn%2FFg1X7Cms4NFLt1wpdFl6RYXmyOL%2BcXuP7XRE4cNrvGDKUTbiaOO7kK1z%2BuGro1qTkySjVbKq3hUsLBnkHq86cBd%2F3juJSvMibY92jqs5c3yb6rSDvld8Ysy0N7hJr9WDQ4wGa3LyjTeZIf8ng1TvH446t7tQeEvoFJEMS5i9HZcMtailTqpIWCvGms0vzEy9Og61b1dn39v%2FVZagDQuTfvNvrySf%2FcGYXrTX6uoH3WnozyRTTD%2BxrwYD3zQs%2Ffml4SuXympskWziXzMjYZ46k4v%2Bw6NClCfbSifrJvS2YGHYlAI%2BMlylLZIP0PAW6ScqJJi5lxzR4dZnkdULfmS; connect.sid=s%3AfnnqPb6g110TGa491jJfOcbuJaPPfAuI.ZlxoOV8%2FIdXd4SlBk%2FOPhcczHInGfx0kjUlRjUX8wxY; 2791.vst=%7B%22s%22%3A%226642b2dc-f63e-4cc7-b27d-154dc6d89b88%22%2C%22t%22%3A%22new%22%2C%22lu%22%3A1602163131793%2C%22lv%22%3A1602163131793%2C%22lp%22%3A0%7D; _fbp=fb.2.1602163133978.539100741; JourneyID=0b2553b7-e9c1-4b3d-893b-1bcb63be21ea; kampyle_userid=eab6-7ed5-b6ec-7f16-e14c-1ca9-ad30-d500; mdigital_alternative_uuid=0e0e-1a51-dc0f-0e38-6fac-254f-051a-b3dc; kampyleUserSession=1602163137010; kampyleUserSessionsCount=1; OptanonAlertBoxClosed=2020-10-08T13:18:58.327Z; OptanonConsent=isIABGlobal=false&datestamp=Thu+Oct+08+2020+14%3A18%3A59+GMT%2B0100+(British+Summer+Time)&version=6.0.0&landingPath=NotLandingPage&groups=1%3A1%2C2%3A1%2C3%3A1%2C4%3A1%2C0_256505%3A1%2C0_256503%3A1%2C0_25576
2

You could try

    mutate { split => { "message" => "
" } }
    ruby {
        code => '
            event.get("message").each { |x|
                m = x.scan(/([-a-zA-Z0-9]+): (.*)/)
                event.set(m[0][0], m[0][1])
            }
        '
    }

Yes, that is a literal newline in the mutate+split. It does not use a regexp.

1 Like
3

thanks Badger. It was a nightmare. But after quite heavy time spend, I found a way

mutate {
    gsub => [ "message", "[\\\\]r", "##" ]
    gsub => [ "message", "[\\\\]n", "##" ]
  }
    kv {
        source => "message"
        value_split_pattern => ":"
        field_split_pattern => "####"
    }
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.