Issue in parsing a custom apache log file with grok plugin

Elastic Stack Logstash
1

Hello,
I'm new to Elastic stack so be patient :slight_smile:

My first purpose is to parse a custom Apache2 access log.
"Custom" means that I only have added a field to the Apache2 Combined Log format.
This filed, added for performance monitoring, is numeric and represent the time taken to serve the request, in microseconds (see "%d" in http://httpd.apache.org/docs/current/mod/mod_log_config.html#customlog.

Without this filed, the log is parsed correctly using the pattern %{COMBINEDAPACHELOG}.

Since COMBINEDAPACHELOG is equal to "HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}", and my custom filed is located before the "referre" filed, I've tryed to parse my custom log with: "HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{NUMBER:reqtime} %{QS:referrer} %{QS:agent}" where reqtime is my numeric new field.

*** This results in a grok error ***

An example line to parse is:
62.101.84.10 - - [11/Apr/2019:16:39:00 +0200] "POST /SedaOnlineMultiWS/rs/mandato/revoca HTTP/1.1" 200 137 180388 "-" "Apache-HttpClient/4.3.1 (java 1.5)"

where "180388" is the value of the added field.

Why i get an error?

Thank you in advance
Regards

Stefano

2 
    grok {
        pattern_definitions => { "HTTPD_COMBINEDLOG" => "%{HTTPD_COMMONLOG} %{NUMBER:reqtime} %{QS:referrer} %{QS:agent}" }
        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
    }

works just fine for me

      "agent" => "\"Apache-HttpClient/4.3.1 (java 1.5)\"",
      "bytes" => "137",
    "request" => "/SedaOnlineMultiWS/rs/mandato/revoca",
       "verb" => "POST",
"httpversion" => "1.1",
   "referrer" => "\"-\"",
   "clientip" => "62.101.84.10",
   "response" => "200",
    "reqtime" => "180388"
3

With your config I obtain the following output:

{
"offset" => 0,
"beat" => {
"name" => "Srv02ficlestack.servizi.infogroup.it",
"hostname" => "Srv02ficlestack.servizi.infogroup.it",
"version" => "6.7.1"
},
"source" => "/opt/elastic/esempi/lineaunica",
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure",
[2] "_geoip_lookup_failure"
],
"@version" => "1",
"message" => "62.101.84.10 - - [11/Apr/2019:16:39:00 +0200] "POST /SedaOnlineMultiWS/rs/mandato/revoca HTTP/1.1" 200 137 18038
"input" => {
"type" => "log"
},
"log" => {
"file" => {
"path" => "/opt/elastic/esempi/lineaunica"
}
},
"@timestamp" => 2019-04-12T14:41:29.702Z,
"prospector" => {
"type" => "log"
},
"host" => {
"name" => "Srv02ficlestack.servizi.infogroup.it"
}
}

My config file is:

input
{
beats
{
port => "5044"
}
}

filter
{
grok {
pattern_definitions => { "HTTPD_COMBINEDLOG" => "%{HTTPD_COMMONLOG} %{NUMBER:reqtime} %{QS:referrer} %{QS:agent}" }
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
geoip
{
source => "clientip"
}
}
output
{
stdout { codec => rubydebug }
}

4

That message does not have referrer or agent fields.

5

A wrong cut&paste, sorry.
Correct output below

{
"offset" => 0,
"beat" => {
"name" => "Srv02ficlestack.servizi.infogroup.it",
"hostname" => "Srv02ficlestack.servizi.infogroup.it",
"version" => "6.7.1"
},
"source" => "/opt/elastic/esempi/lineaunica",
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure",
[2] "_geoip_lookup_failure"
],
"@version" => "1",
"message" => "62.101.84.10 - - [11/Apr/2019:16:39:00 +0200] "POST /SedaOnlineMultiWS/rs/mandato/revoca HTTP/1.1" 200 137 18038
8 "-" "Apache-HttpClient/4.3.1 (java 1.5)"",
"input" => {
"type" => "log"
},
"log" => {
"file" => {
"path" => "/opt/elastic/esempi/lineaunica"
}
},
"@timestamp" => 2019-04-12T14:41:29.702Z,
"prospector" => {
"type" => "log"
},
"host" => {
"name" => "Srv02ficlestack.servizi.infogroup.it"
}
}

6

Does dissect work for you?

dissect { mapping => { "message" => '%{clientip} %{auth} %{ident} [%{timestamp}] "%{verb} %{request} HTTP/%{httpversion}" %{response} %{bytes} %{reqtime} "%{referrer}" "%{agent}"' } }
7

I found the reason!
I sayd that I'm new to this technology end ... in fact ... i didi't started logstash whith "--config.reload.automatic" option so every change I made to config file wasn't effective .

Sorry for your time wasted and thank you so much.

Kind regards

Stefano

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.