Logstash parsing problem

Elastic Stack Logstash
1

Hello guys!

I'm trying to parse the following type of log message:

111.22.333.444 - - [08/Jan/2020:11:50:15 +0100] [https://awdasfe.asfeaf.cas:111] "POST /VFQ3P/asfiheasfhe/v2/safiehjafe/check HTTP/1.1" 204 0 "-" "-" (rt=0.555 urt=0.555 uct=0.122 uht=0.11)

My logstash conf file:

input {
  beats {
    port => 5044
  }
}

filter {
  grok { match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \[%{NOTSPACE:referrer}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" } }

  geoip { source => "clientip" }
}


output {
  elasticsearch {
   hosts => ["localhost:9200"]
   index => "my_index5"
  }
}

I'm using almost the same patterns like in the github pattern library for COMMONAPACHELOG. When I put the code through grok debugger in Kibana it works the way I want but when I try to execute it on machine logstash throws me an error that there is a symbol expected before the "(?:%{WORD:verb} part and when I add there \ there is still a problem.

Does anyone have any suggestions for solving the problem?

Thanks in advance!

2

You need to escape both of the double quotes inside the pattern using backslash, or else use single quotes around the pattern.

grok { match => { "message" => '%{IPORHOST:clientip} ...' } }

If you still get errors then show us the error message.

3

Thank you for your help!

The thing that is bothering me now is this part:

(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})

I think at some point it should show the raw request part like POST /VFQ3P/asfiheasfhe/v2/safiehjafe/check HTTP/1.1 and parse into verb and http version at the same time but I dont have the field rawrequest.

I'd like do it with referrer part as well so I could parse https://awdasfe.asfeaf.cas:111 for fields:

protocol: https
domain1: awdasfe
domain2: asfeaf
domain3: cas
port: 111
4

No, alternation (the pipe character) does not work that way. If tries to parse it using the first part of the pattern (verb/request/httpversion) and only if that fails will it parse using the second part.

If you want to have both then change the initial grok to

%{DATA:rawrequest})

and add a second grok that does

match => { "rawrequest" => "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?" }

The same would apply to the referrer.

5

Thank you very much for replying!

I just found the related topic to my question! How can i parse one field further into different fields in another grok pattern?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.