Logstash grok nginx log

Elastic Stack Logstash
1

Hi dears,
I want to grok nginx access log but i cant get the seperated field of the message ,
i tried to use overwrite and doesn't make the solution.

my Log format is below :
</>
Jun 10, 2021 @ 11:07:43.648
host:logstash.cluster
@timestamp:Jun 10, 2021 @ 11:07:43.648
tags:
_grokparsefailure
message:
52.61.52.88:64539 - - [02/Jun/2021:03:27:35 +0200] "GET https://test.al-domain/public/blank.html HTTP/1.1" 200 1745 "https://test.al-domain/modern/email/Inbox/conversation/2988" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "65.35.42.14:8443" "65.35.42.14:443"
@version:1
path:/var/local/znginx_log.log
_id:jT1k9XkB9s8XMY09VrUf
_type:
_doc
_index:znginx_1
_score:

</>
filter {
grok {
match => { "message" => ["%{IPV4:ip_remote}:%{INT:port} - (?:%{DATA:Domain}\x5C%{USERNAME:user_name}|%{USERNAME:username}@%{DATA:Domain}|-) [%{TIMESTAMP_ISO8601:date_log} %{BASE10NUM:plustime}] %{DATA:req_method} %{DATA:request} HTTP/%{BASE10NUM:httpversion} %{BASE10NUM:response} (?:%{BASE10NUM:bytes}|-) (?:(?:%{URI:referrer}|-)|%{QS:referrer})(?:;|) %{DATA:agent_access} %{NOTSPACE:host} %{NOTSPACE:xff_clientip}" ] }

    overwrite => ["message"]
  }
  date {
match => ["timestamp", "yy-MM-dd HH:mm:ss"]
target => "@timestamp"
timezone => "Europe/Moscow"
}
 mutate {
remove_field => [ "timestamp" ]
}

}
</>

2

Hi,

your GROK pattern does not work for me...

I tried this (based on COMBINEDAPACHELOG)

%{IPORHOST:clientip}:%{NUMBER:client_port} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:host} %{QS:xff_clientip}

Which resulted in


{
  "request": "https://test.al-domain/public/blank.html",
  "agent": "\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36\"",
  "auth": "-",
  "ident": "-",
  "verb": "GET",
  "xff_clientip": "\"65.35.42.14:443\"",
  "client_port": "64539",
  "referrer": "\"https://test.al-domain/modern/email/Inbox/conversation/2988\"",
  "response": "200",
  "bytes": "1745",
  "clientip": "52.61.52.88",
  "host": "\"65.35.42.14:8443\"",
  "httpversion": "1.1",
  "timestamp": "02/Jun/2021:03:27:35 +0200"
}
3

Hi, thanks for replying

I want to get separated fields of the message,
in kibana i recive the message as one field, and whene i try to grok on Dev tools kibana it work but does't sended as it from logstash
Capture

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.