# NGINX logs - Provided Grok expressions do not match field value **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 18, 2018, 11:04am UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518 "2018-04-18T11:04:49Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![astropanic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/astropanic/32/20726_2.png) [@astropanic](https://discuss.elastic.co/u/astropanic)\ **Post date:** [April 18, 2018, 11:04am UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/1 "2018-04-18T11:04:49Z")
I'm trying to write a grok filter for my nginx log. the filter: ``` filter { if [fileset][module] == "nginx" { if [fileset][name] == "access" { grok { match => { "message" => '%{IPV4} - - \[%{HTTPDATE:timestamp}\]\" "%{WORD:method} %{URIPATH:uri_path}(?:%{URIPARAM:uri_params})? HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}" "%{DATA:clientip}" "%{DATA:session}" "%{NUMBER:responsetime}" %{DATA:domain}' } } mutate { convert => ["response", "integer"] convert => ["bytes", "integer"] convert => ["responsetime", "float"] } } } } ``` it works, when I test it with the stdin input. But when I use the beats input on port 5044, I see strange things in kibana: ``` Provided Grok expressions do not match field value: [192.168.32.98...... ``` How to debug what is the issue ? Here a sample line from my nginx logs: ``` 192.168.32.98 - - [18/Apr/2018:11:36:51 +0200]" "GET /?ping HTTP/1.1" 200 63718 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "194.76.219.19" "3294aeff1d3031a4c880ac7497688473" "0.105" "www.example.com" ``` ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [April 18, 2018, 11:30am UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/2 "2018-04-18T11:30:00Z")
Which version of NGINX are you using? Do you have special `log_format` option in your NGINX config? ---
**Author:** ![astropanic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/astropanic/32/20726_2.png) [@astropanic](https://discuss.elastic.co/u/astropanic)\ **Post date:** [April 18, 2018, 11:48am UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/3 "2018-04-18T11:48:39Z")
Yes, I have a custom log format, that's why I'm using my custom pattern matching in the pasted grok config. As I already have written: it works when I testing it with the stdin plugin, It works as well in the grok debugger. Only when using the beats input, I got parsing errors visible in kibana. The log format: ``` log_format main '$remote_addr - $remote_user [$time_local]" "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$cookie_client_id" "$request_time" "$host"'; ``` ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [April 18, 2018, 12:02pm UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/4 "2018-04-18T12:02:57Z")
Are you using the NGINX Filebeat module? If yes, it's possible that the error message is coming from the module. Messages coming from Filebeat go through the default modules pipeline. That's not appropriate in your case. I would customize the existing NGINX pipeline to match your log format. Load it to Elasticsearch with a new ID. Then set the option `pipeline` of the input to use it. ```auto - type: log enabled: true paths: - {{ path/to/your/log }} pipeline: {{ custom-pipeline-id }} ``` Thus, logs coming from this input would go through the customized pipeline. ---
**Author:** ![astropanic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/astropanic/32/20726_2.png) [@astropanic](https://discuss.elastic.co/u/astropanic)\ **Post date:** [April 18, 2018, 12:04pm UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/5 "2018-04-18T12:04:51Z")
I don't get it. And please bear with me, I read all the docs, but feel quite lost. I just want to pipe my logs from nginx through logstash to ES and be able to visualise in a kibana dashboard. Should I abandon the usage of filebeat in my case? Or what are the steps to achieve my goal? ---
**Author:** ![astropanic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/astropanic/32/20726_2.png) [@astropanic](https://discuss.elastic.co/u/astropanic)\ **Post date:** [April 18, 2018, 1:06pm UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/6 "2018-04-18T13:06:23Z")
I got it working now. I removed the filebeat nginx module, and I'm using filebeat only, now it works flawlessly. Is there a way to use the filebeat nginx module with a custom nginx log format? ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [April 18, 2018, 2:16pm UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/7 "2018-04-18T14:16:53Z")
I am glad you got it working. Filebeat modules are made for smoothing the getting started experience of new users. So by default its pipeline does not support custom log formats. However, advanced Filebeat users can modify the patterns in `module/nginx/access/ingest/default.json` and their own format. After the pipeline is updated, it needs to be loaded to the Ingest node. Right now in order to update an existing pipeline, you need to delete it manually and then load the new version. But in the next release `filebeat.update_pipelines` is introduced. If it's set to `true` pipelines are always updated. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 16, 2018, 2:16pm UTC](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/8 "2018-05-16T14:16:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.