_grokparsefailure exception from logstash while stashing events

[RamyaGowda (Ramya)]

Hi team,

I am trying to parse web server logs , i have return configuration file it looks as bellow

input {
file {
path => "/var/log/nginx/access.log"
type => "nginx-access"
sincedb_path => "/var/log/sincedb"
start_position => "beginning"
}
}

filter {
if [type] == "nginx-access" {
grok{
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:time}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} (?:-|%{NUMBER:bytes}) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime}'
}

}
mutate {
convert => { "bytes" => "integer"}
convert => { "status" => "integer"}
convert => { "responseTime" => "float"}
}

geoip {
source => "clientip"
}

useragent {
source => "agent"
target => "useragent"
}
date {
match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}
}

}

output {
if "_grokparsefailure" not in [tags]
{
elasticsearch {
hosts => ["localhost:9200"]
index => "nginx_log_data"
}
}
stdout { codec => rubydebug }
}

sample logs are :

127.0.0.1 - - [25/Jul/2018:12:14:51 +0530] "GET /favicon.ico HTTP/1.1" 404 324 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" 0.000 -

what's wrong with my configuration , why it is throwing exception? kindly help me.

when i am trying it manualy means from grok debugger, pattern and input matching perfectly.

[Krunal_kalaria]

Hello @RamyaGowda,

Can you tell me you have installed x-pack on elastic stack ?

What Elastic stack version are you currently using on which OS ?

Thanks & Regards,
Krunal.

[NerdSec (Nachiket)]

@RamyaGowda

} [%{HTTPDATE:time}]

You have not escaped the "" characters. Could you try something like this?

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time}\] %{GREEDYDATA:data}

This should work. Also, it could be better if you use dissect instead of grok. Just a suggestion. Have a look.

PS: Use the Grok Debugger shipped with Kibana to troubleshoot grok, it is much easier.

[RamyaGowda (Ramya)]

Can you tell me you have installed x-pack on elastic stack ?
No

What Elastic stack version are you currently using on which OS ?

Elastic stack version is 6.x , i am using it in Linux

[RamyaGowda (Ramya)]

@NerdSec

If i use GREEDYDATA after matching HTTPDATE rest of the things considered as single event i don't want it to be like that

[NerdSec (Nachiket)]

It was just a reference to show the issue. But here is the full string:

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} (?:-|%{NUMBER:bytes}) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime}