Hi,
I'd tried to parse NGINX Access log whose log format is '$http_x_forwarded_for - $http_referer - [$time_local] "$request" $status $body_bytes_sent'.
The logs are like below:
- - - - [08/Oct/2020:11:05:50 +0900] "HTTP GET /" 400 157\n"
- - - - [08/Oct/2020:11:05:51 +0900] "HTTP GET /" 304 122\n"
- - - - [08/Oct/2020:11:05:52 +0900] "HTTP GET /" 200 200\n"
- - - - [08/Oct/2020:11:05:53 +0900] "HTTP GET /" 400 157\n"
- - - - [08/Oct/2020:11:05:54 +0900] "HTTP GET /" 304 128\n"
then I'd set my grok filter like below :
match => { "data" => "(?:(?:%{IPORHOST:http_x_forwarded_for}|-) - (?:%{URI:http_referer}|-) - \[%{HTTPDATE:timestamp}\] \\\"HTTP %{WORD:method} \/\\\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)\\n.*)" }
but I've got grokparsefailure.
Raw log from Nginx docker likes this:
{"log":"- - - - [08/Oct/2020:11:05:50 +0900] \"HTTP GET /\" 400 157\n","stream":"stdout","time":"2020-10-08T02:05:50.481864709Z"}
Logstash error is like below
[2020-10-14T12:59:34,833][DEBUG][logstash.pipeline ] output received {"event"=>{"app_id"=>"myapp-web-latest-75f4d67dff-42qd5", "app"=>"app", "dc"=>"kor", "image"=>["app/myapp-web", "latest"], "registry"=>"private-registry.myrepo", "image_tag"=>"latest", "kubernetes"=>{"pod_name"=>"myapp-web-latest-75f4d67dff-42qd5", "host"=>"kube-worker04", "container_name"=>"myapp-web", "namespace_name"=>"app", "container_image"=>"private-registry.myrepo/app/myapp-web:latest"}, "http_method"=>"%{method}", "app_name"=>"app-webui", "@version"=>"1", "instance_num"=>"PROC", "namespace"=>"app", "log_type"=>"RTR", "http_status"=>"%{response}", "cluster"=>"blabla.lablab.com", "data_image"=>"private-registry.private/myapp/myapp-web:latest", "XFF_IP"=>"%{http_x_forwarded_for}", "@timestamp"=>2020-10-14T03:59:34.731Z, "time"=>"2020-10-14T03:59:30.868062327Z", "data"=>"- - - - [14/Oct/2020:12:59:30 +0900] \"HTTP GET /\" 400 157\n", "app_sub"=>"web", "env"=>"dev", "image_name"=>"app/myapp-web", "log"=>"- - - - [14/Oct/2020:12:59:30 +0900] \"HTTP GET /\" 400 157\n", "node_name"=>"kube-worker04", "tags"=>["_grokparsefailure"], "referer"=>"%{http_referer}", "_@timestamp"=>1602647970.868062, "body_bytes_sent"=>"%{bytes}"}}
Need help.
Thanks in advance.