Having trouble ingesting a ModSsecurity log VIA TCP..sort of

Hey guys.. I'm really hoping that someone here can help me out.
I'm working with a Sophos XG firewall that uses Modsecurity as the reverse proxy WAF engine.
Because of support contract restrictions and also the locked down middle-ware we cant get root ( or should I say, aren't allowed to have root ) to modify the system to install a beat or even tweak the behavior of the logging on the device.
I have been able to do a simple tail on a "reverseproxy.log" and pipe it through netcat over to my logstash instance.. this is great work around to get the log into logstash and then up to ES for searching and storage, but the main issue I have is that the record is stored as one big fat message. none of the objects are broken out..
That makes it painful to search and difficult to read.
I had found a modsecurity audit log ingestor on github crafted by "bitsofinfo" https://github.com/bitsofinfo/logstash-modsecurity
I have it configured to accept my tcp stream from the tail of the log on the device.
Whenever a record is sent to the pipeline I see an update in Kibana that says "failed to find message".
I see he is using ruby to split the message but for some reason the message I'm piping in isn't being seen properly.
My data stream looks like this: [Fri Aug 30 08:24:29.915330 2019] timestamp="1567167869" srcip="x.x.x.x" localip="x.x.x.x" user="-" host="x.x.x.x" method="POST" statuscode="200" reason="-" extra="-" exceptions="-" duration="16442" url="/urlto/webpage/or resource" server="server.domian.com" referer="https://whatever.domain.com/default.aspx?Input=whatever123" cookie="User=JSchmoe; Role=IT; UWID=0; TAID=0;ASP.NET_SessionId=gibberishrandomtoken232dsf; set-cookie="-" recvbytes="2047" sentbytes="368" protocol="HTTP/1.1" ctype="application/json" uagent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" querystring="" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" ruleid="31"
...something like that. Anyway I'm trying get this message broken down to expose all of the objects to be searchable, right now the entire glob is in the message and I cant pick at the individual items.. suppose I wanted to build visualization on the number of hits on a Rule ID # or find what the top 5 status codes were. I cant until that's all broken out..
I see he is using ruby to split it up but I cant understand why it's not working?

    ruby {
  code => "
      if !event.get('message').nil?
          modSecSectionData = event.get('message').split(/(?:--[a-fA-F0-9]{8}-([A-Z])--)/)
          for i in 0..((modSecSectionData.length-1)/2)
              sectionName = 'rawSection'.concat(modSecSectionData.shift)
              sectionData = modSecSectionData.shift
              sectionName = sectionName.strip
              if !sectionData.nil?
                  sectionData = sectionData.strip
              event.set(sectionName, sectionData)


Truth be told I'm real new to Ruby, Logstash and ES. But unfortunately this was dropped on my desk and I am "supposed" to have a solution by first A.M. when we return after the holiday weekend..
I'm hoping someone can help me out here, I would be exceptionally grateful.

Assuming that you accidentally deleted a " when redacting the ASP session cookie you can parse this using

kv {}

which will get you

            "reason" => "-",
        "statuscode" => "200",
            "uagent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36",
              "user" => "-",
           "localip" => "x.x.x.x",
           "referer" => "https://whatever.domain.com/default.aspx?Input=whatever123",
            "cookie" => "User=JSchmoe; Role=IT; UWID=0; TAID=0;ASP.NET_SessionId=gibberishrandomtoken232dsf;",
          "protocol" => "HTTP/1.1",
            "ruleid" => "31",
            "method" => "POST",

etc. You will also want to get the date, which you can do using

    dissect { mapping => { "message" => "[%{[@metadata][timestamp]}]%{}" } }
    date { match => [ "[@metadata][timestamp]", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ] }

Whoa! Wait a minute...
So I dont need this huge convoluted filter to do this?
just a cute little kv {}?!
kv {}
dissect { mapping => { "message" => "[%{[@metadata][timestamp]}]%{}" } }
date { match => [ "[@metadata][timestamp]", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ] }
I tried it and yes it does a much better job of breaking it apart.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.