Need Robust Grok filter for nginx error log format

Elastic Stack Logstash
1

We are facing an issue while filtering nginx error log format ... suppose below are two log snippet from the error log

  1. 2020/11/17 13:04:05 [error] 32237#32237: *4185303 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 122.180.250.38, server: 10.222.10.20, request: "GET /favicon.ico HTTP/1.1", host: "niku.vinsupplier.com", referrer: "https://niku.vinsupplier.com/eRetailWeb/SellerPanelBS.action"

  2. 2020/11/17 13:04:05 [error] 32237#32237: *4185303 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 122.180.250.38, server: 10.222.10.20, request: "GET /favicon.ico HTTP/1.1", host: "niku.vinsupplier.com"

And below is the grok pattern

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{DATA:errormessage}, client: %{IP:client}, server: %{IP:server}, request: \"(?<httprequest>%{WORD:httpcommand} %{UNIXPATH:httpfile} HTTP/(?<httpversion>[0-9.]*))\", host: \"%{DATA:host}\", referrer: \"%{DATA:referrer}\"

So this grok pattern fails if any entry comes with 2nd type of snippet i.e. if the referrer part is missing. We want to make it robust so that in case the referrer part is missing it should treat it as null instead of error out. Please help to achieve a proper grok pattern for this requirement?

2

You can make part of a pattern optional by surrounding it with ( )?, so you could try

(, referrer: \"%{DATA:referrer}\")?

I recommend that you anchor your patterns and avoid DATA or GREEDYDATA in favour of cheaper patterns such as NOTSPACE.

3

Thank you so much @Badger for the suggestion. I will definitely try this

4

Hello @Badger

Your trick worked and helped me tweak grok filter to make it more robust. We are not facing any issues since last evening now. Also we are facing a strange issue in access logs where in suppose response body is coming like this

resp_body:"{"responseCode":0,"responseMessage":"Success","totalOrders":10,"totalPages":1,"currentPage":1,"order":[{"order_no":"NYK-56396963-0544196","eretailOrderNo":"NAH43103202","masterOrderNo":"NAH43103202","status":"Shipped complete","remarks":"","grandtotal":"2822.600","createAtStoreDate":"10/11/2020 23:19:04","shippingpkgcount":"2","itemcount":"10.000","shippingaddress":"chapra ..sahebganj..sonarpatti Landmark: near libas mart","mobileno":"8789943063","is_giftwrap":"no","giftwrap_msg":"","giftwrap_charges":"0.000","cancel_date":"","udf1":null,"udf2":null,"udf3":null,"udf4":"COD","udf5":"10","udf6":null,"udf7":null,"udf8":null,"udf9":null,"udf10":null,"pickupLocation":"","extFulFillmentLocCode":"","shipdetail":[{"qty":"1","transporter":"Delhivery Surface","transporterstatusremark":"Handover to co-located facility","transporterstatus":"INTRANSIT","shipdate":"13/11/2020 08:42:05","updated_date":"18/11/2020 07:50:13","delivereddate":"","refereceNo":"NBL320254620","wh_Loccode":"NBL","wh_Locname""

But we are not able to capture it via this filter resp_body:\"%{NOTSPACE:resp_body}\"

It is breaking the match and only partial body is being captured. Please help me to capture it full till the end?

5

Hello @grumo35 hope you are doing well, could you please help in this regard?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.