Hi team,
I am trying to parse web server logs , i have return configuration file it looks as bellow
input {
file {
path => "/var/log/nginx/access.log"
type => "nginx-access"
sincedb_path => "/var/log/sincedb"
start_position => "beginning"
}
}
filter {
if [type] == "nginx-access" {
grok{
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:time}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} (?:-|%{NUMBER:bytes}) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime}'
}
}
mutate {
convert => { "bytes" => "integer"}
convert => { "status" => "integer"}
convert => { "responseTime" => "float"}
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
date {
match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}
}
}
output {
if "_grokparsefailure" not in [tags]
{
elasticsearch {
hosts => ["localhost:9200"]
index => "nginx_log_data"
}
}
stdout { codec => rubydebug }
}
sample logs are :
127.0.0.1 - - [25/Jul/2018:12:14:51 +0530] "GET /favicon.ico HTTP/1.1" 404 324 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" 0.000 -
what's wrong with my configuration , why it is throwing exception? kindly help me.
when i am trying it manualy means from grok debugger, pattern and input matching perfectly.
Hello @RamyaGowda ,
Can you tell me you have installed x-pack on elastic stack ?
What Elastic stack version are you currently using on which OS ?
Thanks & Regards,
Krunal.
NerdSec
(Nachiket)
July 25, 2018, 8:47am
3
RamyaGowda:
} [%{HTTPDATE:time}]
You have not escaped the " " characters. Could you try something like this?
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time}\] %{GREEDYDATA:data}
This should work. Also, it could be better if you use dissect instead of grok. Just a suggestion. Have a look.
PS: Use the Grok Debugger shipped with Kibana to troubleshoot grok, it is much easier.
Can you tell me you have installed x-pack on elastic stack ?
No
What Elastic stack version are you currently using on which OS ?
Elastic stack version is 6.x , i am using it in Linux
@NerdSec
If i use GREEDYDATA after matching HTTPDATE rest of the things considered as single event i don't want it to be like that
NerdSec
(Nachiket)
July 25, 2018, 9:06am
6
It was just a reference to show the issue. But here is the full string:
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} (?:-|%{NUMBER:bytes}) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime}
system
(system)
Closed
August 22, 2018, 9:06am
7
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.