Any filter for parsing the URL request from firewalls?

regarding payload within firewall logs, the request field has a single line separated by \r\n which I've made it into individual line. Any known parsers/filter to make this into key-value pairs?
(especially the User-Agent , host etc all into individual fields?) and then the cookie into individual fields as well

Please note below example, I've split lines to make it readable, but it is all into one-line with \r\n as field splitter.

Host: www.something.co.uk
Connection: keep-alive
User-Agent: Mozilla/5.0 (Linux; Android 8.1.0; DUB-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36
Accept: image/webp,image/apng,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://www.something.co.uk/assets/styles.css?20201008.2
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: check=true; AMCVS_BB2A12535131457C0A490D45%40AdobeOrg=1; AMCV_BB2A12535131457C0A490D45%40AdobeOrg=-1712354808%7CMCIDTS%7C18544%7CMCMID%7C01238256657182159320825076497090008339%7CMCAAMLH-1602767893%7C6%7CMCAAMB-1602767893%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1602170294s%7CNONE%7CMCAID%7CNONE%7CvVersion%7C4.3.0; mboxEdgeCluster=37; buyapowa_voucher_code=undefined; channeloriginator=seo; channelcloser=seo; VF-BINS=c7b152e2-4c0f-49aa-abe3-e657dc85e224; VF-SID=2748be33-f576-4248-a8b9-57639b73a394; VF-OPT=true; VF-UUID=42f94982-936e-4be1-a498-80a11195e9c2; _uetsid=cc30e5d0096811eb85c1e5508f491fe9; _uetvid=cc3342d0096811eba4063f93798707ac; PlatformAuthToken=PiwBWOulJRn%2FxEjZszyVdTPxfwioN5VzlOYYBTZMXj74iUXWOibanjacplhKwSNciIbBv2WMG8uANwiktrJe%2BLAhWkrm1SUkx2XfkctsBzPG7mqrkperzAbTdjnn%2FFg1X7Cms4NFLt1wpdFl6RYXmyOL%2BcXuP7XRE4cNrvGDKUTbiaOO7kK1z%2BuGro1qTkySjVbKq3hUsLBnkHq86cBd%2F3juJSvMibY92jqs5c3yb6rSDvld8Ysy0N7hJr9WDQ4wGa3LyjTeZIf8ng1TvH446tLHJVEmZBb0t%2FdM7sD0CUHsbrm%2FKa%2FS27LPCg9Ur4mb4hbmi8G3XOsVkeyYM2hEegeP%2B8a88eaa9da11daff849648423eb5d80c32eeed61570069d77a4edfd1; TS01056ccb030=01f849ee050c7e8614acb5e5278b840d830f072001bd7c6e633ebe76e6eebaf9ab435ceeae04fd395a3b1076a368d2cc7594b2e36e; _pin_unauth=dWlkPU1EaGlNelpsWVRRdFpHRm1OaTAwTkdSbUxXRmtZamN0TldVMFl6QXlObU00TVRsaA; AuthenticatedTracker=1602166729; Session=j%3A%7B%22platformSessionId%22%3A%220b2553b7-e9c1-4b3d-893b-1bcb63be21ea%22%2C%22subscriptionIdHash%22%3A%22f9a5a55a396eececfb97b49b%22%2C%22subscriptionType%22%3A%22something%22%2C%22assuranceLevel%22%3A2%2C%22givenName%22%3A%22SULTANA%20MUNNI%22%2C%22numberOfAccounts%22%3A1%2C%22numberOfSubscriptions%22%3A1%2C%22accountCategory%22%3A%22Individual%22%2C%22accountSubCategory%22%3A%22Consumer%22%7D; PlatformAccessToken=PiwBWOulJRn%2FxEjZszyVdTPxfwioN5VzlOYYBTZMXj74iUXWOibanjacplhKwSNciIbBv2WMG8uANwiktrJe%2BLAhWkrm1SUkx2XfkctsBzPG7mqrkperzAbTdjnn%2FFg1X7Cms4NFLt1wpdFl6RYXmyOL%2BcXuP7XRE4cNrvGDKUTbiaOO7kK1z%2BuGro1qTkySjVbKq3hUsLBnkHq86cBd%2F3juJSvMibY92jqs5c3yb6rSDvld8Ysy0N7hJr9WDQ4wGa3LyjTeZIf8ng1TvH446t7tQeEvoFJEMS5i9HZcMtailTqpIWCvGms0vzEy9Og61b1dn39v%2FVZagDQuTfvNvrySf%2FcGYXrTX6uoH3WnozyRTTD%2BxrwYD3zQs%2Ffml4SuXympskWziXzMjYZ46k4v%2Bw6NClCfbSifrJvS2YGHYlAI%2BMlylLZIP0PAW6ScqJJi5lxzR4dZnkdULfmS; connect.sid=s%3AfnnqPb6g110TGa491jJfOcbuJaPPfAuI.ZlxoOV8%2FIdXd4SlBk%2FOPhcczHInGfx0kjUlRjUX8wxY; 2791.vst=%7B%22s%22%3A%226642b2dc-f63e-4cc7-b27d-154dc6d89b88%22%2C%22t%22%3A%22new%22%2C%22lu%22%3A1602163131793%2C%22lv%22%3A1602163131793%2C%22lp%22%3A0%7D; _fbp=fb.2.1602163133978.539100741; JourneyID=0b2553b7-e9c1-4b3d-893b-1bcb63be21ea; kampyle_userid=eab6-7ed5-b6ec-7f16-e14c-1ca9-ad30-d500; mdigital_alternative_uuid=0e0e-1a51-dc0f-0e38-6fac-254f-051a-b3dc; kampyleUserSession=1602163137010; kampyleUserSessionsCount=1; OptanonAlertBoxClosed=2020-10-08T13:18:58.327Z; OptanonConsent=isIABGlobal=false&datestamp=Thu+Oct+08+2020+14%3A18%3A59+GMT%2B0100+(British+Summer+Time)&version=6.0.0&landingPath=NotLandingPage&groups=1%3A1%2C2%3A1%2C3%3A1%2C4%3A1%2C0_256505%3A1%2C0_256503%3A1%2C0_25576

You could try

    mutate { split => { "message" => "
" } }
    ruby {
        code => '
            event.get("message").each { |x|
                m = x.scan(/([-a-zA-Z0-9]+): (.*)/)
                event.set(m[0][0], m[0][1])
            }
        '
    }

Yes, that is a literal newline in the mutate+split. It does not use a regexp.

1 Like

thanks Badger. It was a nightmare. But after quite heavy time spend, I found a way

mutate {
    gsub => [ "message", "[\\\\]r", "##" ]
    gsub => [ "message", "[\\\\]n", "##" ]
  }
    kv {
        source => "message"
        value_split_pattern => ":"
        field_split_pattern => "####"
    }