Troubles with apache rewrite log grok

Elastic Stack Logstash
1

Hi!

First time posting, I haven't had much luck finding apache rewrite log info anywhere online and I've been struggling with this so I figured I'd check withe the community.

I'm new to logstash (6.3.2) and this is my first post here.

Here is my grok syntax:

if [type] in [ "apache_rewrite" ] {
   grok {
      match => {
         "message" => "%{IPORHOST:client_ip} %{USER} %{USER:http_user} \[%{HTTPDATE}\] %{SYSLOG5424SD}%{SYSLOG5424SD} \(%{NOTSPACE:loglevel}\) %{GREEDYDATA:message}"
      }
      remove_field => ["message"]
   }
}

I'm trying to match the "The RewriteLog log file format is as follows" (ctrl+f that here: apache 2.2 documentation

Here is an example log entry:

10.127.4.53 - - [31/May/2019:03:25:36 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) go-ahead with proxy request proxy:balancer://something/something/processSM.do [OK]
10.127.33.37 - - [31/May/2019:03:26:02 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) pass through /server-status
10.127.33.37 - - [31/May/2019:03:26:02 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) pass through /server-status
10.127.33.37 - - [31/May/2019:03:26:02 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) pass through /server-status
10.127.33.37 - - [31/May/2019:03:26:02 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) pass through /server-status
10.127.2.52 - - [31/May/2019:03:26:09 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) go-ahead with proxy request proxy:balancer://something/something/process.do [OK]
10.127.4.54 - - [31/May/2019:03:26:09 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) go-ahead with proxy request proxy:balancer://something/something/processSM.do [OK]

I did use grokdebug and its matching my data, however kibana is putting everything in message and isn't splitting the data into fields.

Thoughts anyone? Any feedback would be greatly appreciated. Thanks!

2

That suggests that [type] does not match "apache_rewrite". If it did match and then the grok executed successfully then you would not have a field called message.

A configuration like this

input { generator { count => 1 lines => [ '10.127.33.37 - - [31/May/2019:03:26:02 +0000] [10.127.3.49/sid#7f7e75105ad8][rid#7f7e38002978/initial] (1) pass through /server-status' ] } }
filter {
    grok {
        match => { "message" => "%{IPORHOST:client_ip} %{USER} %{USER:http_user} \[%{HTTPDATE}\] %{SYSLOG5424SD}%{SYSLOG5424SD} \(%{NOTSPACE:loglevel}\) %{GREEDYDATA:message}" }
        remove_field => ["message"]
    }
}
output { stdout { codec => rubydebug { metadata => false } } }

results in

{
      "host" => "example.com",
 "client_ip" => "10.127.33.37",
  "loglevel" => "1",
 "http_user" => "-",
"@timestamp" => 2019-06-04T19:02:27.541Z,
  "@version" => "1",
  "sequence" => 0
}
3

Hey Thanks for the feedback Badger!

The strange part is that it does match:

file {
    path => "/var/log/httpd/something-rewrite.log"
    type => "apache_rewrite"
}

I think this may call for a lengthy investigation into logs after turning on debug logging

5

I spun up an local ELK docker env and everything works fine. Something must be up with what I'm doing via AWS ES / Kibana. It's probably something silly on my end. Case closed!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.