Hey guys.. I'm really hoping that someone here can help me out.
I'm working with a Sophos XG firewall that uses Modsecurity as the reverse proxy WAF engine.
Because of support contract restrictions and also the locked down middle-ware we cant get root ( or should I say, aren't allowed to have root ) to modify the system to install a beat or even tweak the behavior of the logging on the device.
I have been able to do a simple tail on a "reverseproxy.log" and pipe it through netcat over to my logstash instance.. this is great work around to get the log into logstash and then up to ES for searching and storage, but the main issue I have is that the record is stored as one big fat message. none of the objects are broken out..
That makes it painful to search and difficult to read.
I had found a modsecurity audit log ingestor on github crafted by "bitsofinfo" https://github.com/bitsofinfo/logstash-modsecurity
I have it configured to accept my tcp stream from the tail of the log on the device.
Whenever a record is sent to the pipeline I see an update in Kibana that says "failed to find message".
I see he is using ruby to split the message but for some reason the message I'm piping in isn't being seen properly.
My data stream looks like this: [Fri Aug 30 08:24:29.915330 2019] timestamp="1567167869" srcip="x.x.x.x" localip="x.x.x.x" user="-" host="x.x.x.x" method="POST" statuscode="200" reason="-" extra="-" exceptions="-" duration="16442" url="/urlto/webpage/or resource" server="server.domian.com" referer="https://whatever.domain.com/default.aspx?Input=whatever123" cookie="User=JSchmoe; Role=IT; UWID=0; TAID=0;ASP.NET_SessionId=gibberishrandomtoken232dsf; set-cookie="-" recvbytes="2047" sentbytes="368" protocol="HTTP/1.1" ctype="application/json" uagent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" querystring="" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" ruleid="31"
...something like that. Anyway I'm trying get this message broken down to expose all of the objects to be searchable, right now the entire glob is in the message and I cant pick at the individual items.. suppose I wanted to build visualization on the number of hits on a Rule ID # or find what the top 5 status codes were. I cant until that's all broken out..
I see he is using ruby to split it up but I cant understand why it's not working?
ruby {
code => "
if !event.get('message').nil?
modSecSectionData = event.get('message').split(/(?:--[a-fA-F0-9]{8}-([A-Z])--)/)
modSecSectionData.shift
for i in 0..((modSecSectionData.length-1)/2)
sectionName = 'rawSection'.concat(modSecSectionData.shift)
sectionData = modSecSectionData.shift
sectionName = sectionName.strip
if !sectionData.nil?
sectionData = sectionData.strip
end
event.set(sectionName, sectionData)
end
end
"
}
}
}
Truth be told I'm real new to Ruby, Logstash and ES. But unfortunately this was dropped on my desk and I am "supposed" to have a solution by first A.M. when we return after the holiday weekend..
I'm hoping someone can help me out here, I would be exceptionally grateful.
Thanks!