Grok check if exists

Hi, I have 2 type of nginx logs:

x.x.x.x - - [13/May/2020:16:42:27 +0430] "GET /cache/68/fb/68fb0ff0cf190dc7fcc567e76e1e4c2b.jpg HTTP/1.1" 200 79059

x.x.x.x - - [13/May/2020:16:40:06 +0430] "GET /dl-apk/1585150257_vhTIEU0sjQ.apk?st=hqWJPpX_LbM6J5qEiW0U4w&e=1589385638&product_id=184623&package_name=com.mypackage&user_id=3693549 HTTP/1.1" 200 67244157

This in my grok:

%{IPV4:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATH:request}.*(?:product_id=%{INT:product_id}).*(?:package_name=%{DATA:package_name}&).*(?:user_id=%{INT:user_id}) HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes_sent}

It's valid for second log type but not for the first one.
I guess I could make a if exist condition with (?:) (if product_id exist or does not) but I don't know how.
I see (?:%{INT:number}|-) means number or - character, but what if does not exists ?
Thanx a lot

you can use array of patterns to match

further info in the docs

1 Like

I did it as you suggested:

filter {
                grok {  match => [ "message" , "%{IPV4:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATH:request}.*(?:product_id=%{INT:product_id}).*(?:package_name=%{DATA:package_name}&).*(?:user_id=%{INT:user_id}) HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes_sent}"]  }
                grok {  match => [ "message" , "%{IPV4:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATH:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes_sent}"]  }
                geoip {
                        source => "clientip"
                        target => "geoip"
                mutate {                                                                                                                                 
                      convert => ["response", "integer"]
                      convert => ["bytes_sent", "integer"]

but for both matches logstash add _grokparsefailure to tags.
In general it works. thanx.

you’re running two grok, that’s why you have parse failure. it will match first then have parse failure on the next and vice versa.

you could do

      grok {
        match => {
          "message" => [

however, be careful if you have double quotes in your pattern then you need you wrap the pattern in a single quote. in general, don’t mix them

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.