# Need Robust Grok filter for nginx error log format **URL:** \ **Category:** Logstash\ **Created:** [November 17, 2020, 7:40am UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630 "2020-11-17T07:40:04Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\ **Post date:** [November 17, 2020, 7:40am UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630/1 "2020-11-17T07:40:04Z")
We are facing an issue while filtering nginx error log format ... suppose below are two log snippet from the error log 1. `2020/11/17 13:04:05 [error] 32237#32237: *4185303 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 122.180.250.38, server: 10.222.10.20, request: "GET /favicon.ico HTTP/1.1", host: "niku.vinsupplier.com", referrer: "https://niku.vinsupplier.com/eRetailWeb/SellerPanelBS.action"` 2. `2020/11/17 13:04:05 [error] 32237#32237: *4185303 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 122.180.250.38, server: 10.222.10.20, request: "GET /favicon.ico HTTP/1.1", host: "niku.vinsupplier.com"` And below is the grok pattern `(?%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{DATA:errormessage}, client: %{IP:client}, server: %{IP:server}, request: \"(?%{WORD:httpcommand} %{UNIXPATH:httpfile} HTTP/(?[0-9.]*))\", host: \"%{DATA:host}\", referrer: \"%{DATA:referrer}\"` So this grok pattern fails if any entry comes with 2nd type of snippet i.e. if the referrer part is missing. We want to make it robust so that in case the referrer part is missing it should treat it as null instead of error out. Please help to achieve a proper grok pattern for this requirement? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 17, 2020, 1:58pm UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630/2 "2020-11-17T13:58:39Z")
You can make part of a pattern optional by surrounding it with ( )?, so you could try ``` (, referrer: \"%{DATA:referrer}\")? ``` I recommend that you [anchor](https://www.elastic.co/blog/do-you-grok-grok) your patterns and avoid DATA or GREEDYDATA in favour of cheaper patterns such as NOTSPACE. ---
**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\ **Post date:** [November 17, 2020, 2:36pm UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630/3 "2020-11-17T14:36:03Z")
Thank you so much @Badger for the suggestion. I will definitely try this ---
**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\ **Post date:** [November 18, 2020, 2:39am UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630/4 "2020-11-18T02:39:38Z")
Hello @Badger Your trick worked and helped me tweak grok filter to make it more robust. We are not facing any issues since last evening now. Also we are facing a strange issue in access logs where in suppose response body is coming like this ` resp_body:"{"responseCode":0,"responseMessage":"Success","totalOrders":10,"totalPages":1,"currentPage":1,"order":[{"order_no":"NYK-56396963-0544196","eretailOrderNo":"NAH43103202","masterOrderNo":"NAH43103202","status":"Shipped complete","remarks":"","grandtotal":"2822.600","createAtStoreDate":"10/11/2020 23:19:04","shippingpkgcount":"2","itemcount":"10.000","shippingaddress":"chapra ..sahebganj..sonarpatti Landmark: near libas mart","mobileno":"8789943063","is_giftwrap":"no","giftwrap_msg":"","giftwrap_charges":"0.000","cancel_date":"","udf1":null,"udf2":null,"udf3":null,"udf4":"COD","udf5":"10","udf6":null,"udf7":null,"udf8":null,"udf9":null,"udf10":null,"pickupLocation":"","extFulFillmentLocCode":"","shipdetail":[{"qty":"1","transporter":"Delhivery Surface","transporterstatusremark":"Handover to co-located facility","transporterstatus":"INTRANSIT","shipdate":"13/11/2020 08:42:05","updated_date":"18/11/2020 07:50:13","delivereddate":"","refereceNo":"NBL320254620","wh_Loccode":"NBL","wh_Locname""` But we are not able to capture it via this filter `resp_body:\"%{NOTSPACE:resp_body}\"` It is breaking the match and only partial body is being captured. Please help me to capture it full till the end? ---
**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\ **Post date:** [November 18, 2020, 2:40pm UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630/5 "2020-11-18T14:40:04Z")
Hello @grumo35 hope you are doing well, could you please help in this regard? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 16, 2020, 2:40pm UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630/6 "2020-12-16T14:40:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.