Grok check if exists

[Mohammad_Mousavi]

Hi, I have 2 type of nginx logs:

x.x.x.x - - [13/May/2020:16:42:27 +0430] "GET /cache/68/fb/68fb0ff0cf190dc7fcc567e76e1e4c2b.jpg HTTP/1.1" 200 79059

x.x.x.x - - [13/May/2020:16:40:06 +0430] "GET /dl-apk/1585150257_vhTIEU0sjQ.apk?st=hqWJPpX_LbM6J5qEiW0U4w&e=1589385638&product_id=184623&package_name=com.mypackage&user_id=3693549 HTTP/1.1" 200 67244157

This in my grok:

%{IPV4:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATH:request}.*(?:product_id=%{INT:product_id}).*(?:package_name=%{DATA:package_name}&).*(?:user_id=%{INT:user_id}) HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes_sent}

It's valid for second log type but not for the first one.
I guess I could make a if exist condition with (?:) (if product_id exist or does not) but I don't know how.
I see (?:%{INT:number}|-) means number or - character, but what if does not exists ?
Thanx a lot

[ptamba]

you can use array of patterns to match

further info in the docs

[Mohammad_Mousavi]

I did it as you suggested:

filter {
                grok {  match => [ "message" , "%{IPV4:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATH:request}.*(?:product_id=%{INT:product_id}).*(?:package_name=%{DATA:package_name}&).*(?:user_id=%{INT:user_id}) HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes_sent}"]  }
                grok {  match => [ "message" , "%{IPV4:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATH:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes_sent}"]  }
                geoip {
                        source => "clientip"
                        target => "geoip"
                }
                mutate {                                                                                                                                 
                      convert => ["response", "integer"]
                      convert => ["bytes_sent", "integer"]
                }
                }
       }

but for both matches logstash add _grokparsefailure to tags.
In general it works. thanx.

[ptamba]

you’re running two grok, that’s why you have parse failure. it will match first then have parse failure on the next and vice versa.

you could do

      grok {
        match => {
          "message" => [
            "pattern1",
            "pattern2"
          ]
        }
      }

however, be careful if you have double quotes in your pattern then you need you wrap the pattern in a single quote. in general, don’t mix them