Logstash parsing problem

Hello guys!

I'm trying to parse the following type of log message:

111.22.333.444 - - [08/Jan/2020:11:50:15 +0100] [https://awdasfe.asfeaf.cas:111] "POST /VFQ3P/asfiheasfhe/v2/safiehjafe/check HTTP/1.1" 204 0 "-" "-" (rt=0.555 urt=0.555 uct=0.122 uht=0.11)

My logstash conf file:

input {
  beats {
    port => 5044
  }
}

filter {
  grok { match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \[%{NOTSPACE:referrer}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" } }

  geoip { source => "clientip" }
}


output {
  elasticsearch {
   hosts => ["localhost:9200"]
   index => "my_index5"
  }
}

I'm using almost the same patterns like in the github pattern library for COMMONAPACHELOG. When I put the code through grok debugger in Kibana it works the way I want but when I try to execute it on machine logstash throws me an error that there is a symbol expected before the "(?:%{WORD:verb} part and when I add there \ there is still a problem.

Does anyone have any suggestions for solving the problem?

Thanks in advance!

You need to escape both of the double quotes inside the pattern using backslash, or else use single quotes around the pattern.

grok { match => { "message" => '%{IPORHOST:clientip} ...' } }

If you still get errors then show us the error message.

Thank you for your help!

The thing that is bothering me now is this part:

(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})

I think at some point it should show the raw request part like POST /VFQ3P/asfiheasfhe/v2/safiehjafe/check HTTP/1.1 and parse into verb and http version at the same time but I dont have the field rawrequest.

I'd like do it with referrer part as well so I could parse https://awdasfe.asfeaf.cas:111 for fields:

protocol: https
domain1: awdasfe
domain2: asfeaf
domain3: cas
port: 111

No, alternation (the pipe character) does not work that way. If tries to parse it using the first part of the pattern (verb/request/httpversion) and only if that fails will it parse using the second part.

If you want to have both then change the initial grok to

%{DATA:rawrequest})

and add a second grok that does

match => { "rawrequest" => "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?" }

The same would apply to the referrer.

Thank you very much for replying!

I just found the related topic to my question! How can i parse one field further into different fields in another grok pattern?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.