How to use grok pattern to separate nginx access log fields !?

i want from access log nginx from elk_stack, Of all fields ((timestamp, version, host, clientip, ident, auth, verb, request, httpversion, response, bytes, referrer, agent)) Separately from each field Visualize and make a dashboard, I know this should be done using groke patterns, But I do not know which of the following patterns is right for my work!

grok {
match => { "message" => "%{SYNTAX:SEMANTIC}" }
}


grok {
patterns_dir => "./patterns"
match => [
"message", "%{BASE_PATTERN} %{EXTRA_PATTERN}",
"message", "%{BASE_PATTERN}",
"message", "%{SOME_OTHER_PATTERN}"
]
}


filter {
if [type] == "sometype1" or [type] == "sometype2" {
grok {
match => [ "message", "%{POSINT:moduleid}%{SPACE}%{NUMBER:date}"]
}
}
}


filter {

multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => "previous"
}

if [type] == "A" {
grok {
match => [ any other pattern ...
}
}


filter {
grok {
match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
overwrite => [ "message" ]
}
mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
}
geoip {
source => "clientip"
target => "geoip"
add_tag => [ "nginx-geoip" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
useragent {
source => "agent"
}
+}

It looks like %{COMBINEDAPACHELOG} should work fine. If not please post an example line from your logs. Copy/paste, no screenshot.

U can use grok filter example for nginx + apache in nginx, apache module of filebeat (ingest pipeline).
Or can u post your log. i will help u and share with u my grok filter :slight_smile:

my server log and error log ...

5.5.50.50 - - [15/Jul/2017:10:48:56 +0430] 200 "POST /video-crawler/publish?id=9117166 HTTP/1.1" 54 "https://admin.mydomain.com/video-crawler/list?per-page=240&page=46" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" "-"

or

2017/07/15 10:48:33 [error] 5635#5635: *12621 connect() failed (113: No route to host) while connecting to upstream, client: 5.5.50.50, server: www.mydomain.com, request: "GET /channel/list?q=%D8%A8%D8%AE HTTP/1.1", upstream: "https://172.17.10.21:443/channel/list?q=%D8%A8%D8%AE", host: "admin.mydomain.com", referrer: "https://admin.mydomain.com/video-crawler/list?per-page=240&page=46"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.