# Nginx access and error logs show incorroct value in kibana dashboard **URL:** \ **Category:** Kibana\ **Created:** [April 7, 2021, 11:52am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453 "2021-04-07T11:52:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 7, 2021, 11:52am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/1 "2021-04-07T11:52:12Z")
I am new in ELK. I have ELK stack 7.12.0 (filebeat -\> logstash -\> elasticsearch cluster -\> kibana) I installed filebeat on nginx server and enabled nginx module. these are my configuration files: filebeat.yml ```auto filebeat.inputs: - type: log enabled: true paths: - /var/log/*.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 1 setup.kibana: host: "x.x.x.x:5601" output.logstash: hosts: ["y.y.y.y:5044"] processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` filebeat/modules.d/nginx.yml ```auto - module: nginx access : enabled: true var.paths: ["/var/log/nginx/access.log"] # Error logs error: enabled: true var.paths: ["/var/log/nginx/error.log"] ``` kibana discovery shows corroct data: ![discover](https://us1.discourse-cdn.com/elastic/original/3X/f/7/f7cfbbaa0a350464169b1ed99c0e1cac108b8ca7.png) But in dashboard access graph has no data and error graph shows access data!!!!! ![dashboard](https://us1.discourse-cdn.com/elastic/original/3X/b/a/ba4a637e38d0243022c06e7f16b580e6fd13691c.png) what is your idea? ---
**Author:** ![jportner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jportner/32/75692_2.png) [@jportner](https://discuss.elastic.co/u/jportner)\ **Post date:** [April 7, 2021, 3:28pm UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/2 "2021-04-07T15:28:44Z")
It looks like the Filebeat Nginx module is not parsing the access logs correctly at ingest time -- so when the document is ingested into Elasticsearch, the entire line is showing up in the "message" field, and the other fields are not populated. The [Filebeat Nginx module documentation](https://www.elastic.co/guide/en/beats/filebeat/7.12/filebeat-module-nginx.html) states that it has been tested with Nginx version 1.10. What version of Nginx are you using? Also, can you share a sample of a few lines from your Nginx access log here? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 7, 2021, 4:47pm UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/3 "2021-04-07T16:47:04Z")
Did you run `filebeat setup -e` Or did you just load the dashboards? ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 9, 2021, 5:59pm UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/4 "2021-04-09T17:59:54Z")
Yes, I did it. ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 9, 2021, 6:16pm UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/5 "2021-04-09T18:16:56Z")
nginx version: nginx/1.16.1 And this is the sample of my nginx access log: ![10](https://us1.discourse-cdn.com/elastic/original/3X/0/7/07ea3ba27e5188bc71030e6ad0cc50c4cdeccd7b.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 10, 2021, 2:16am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/6 "2021-04-10T02:16:22Z")
Can you please share your logstash pipeline. ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 10, 2021, 6:09am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/7 "2021-04-10T06:09:17Z")
/etc/logstash/conf.d/02-beats-input.conf ```auto input { beats { host => "0.0.0.0" port => "5044" } } ``` /etc/logstash/conf.d/30-elasticsearch-output.conf ```auto output { elasticsearch { hosts => ["http://X.X.X.X:9200"] manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } ``` And I dont have filtering pipeline. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 10, 2021, 6:13am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/8 "2021-04-10T06:13:04Z")
Here is my beats -\> logstash - es conf Pay special attention the the pipeline reference, what is probably happening is the the ingest pipeline (the ngnix ingest pipeline) is not being called, thus the logs are not parse correctly If you first point filebeat at elasticsearch and run `filebeat setup -e` Then point filebeat to logstash and use this all the ILM, Templates, Pipelines, dashboards etc should work. ``` ################################################ # beats->logstash->es default config. ################################################ input { beats { port => 5044 } } output { if [@metadata][pipeline] { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" pipeline => "%{[@metadata][pipeline]}" user => "elastic" password => "secret" } } else { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" user => "elastic" password => "secret" } } } ``` ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 10, 2021, 6:46am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/9 "2021-04-10T06:46:18Z")
Thank you @stephenb. I run this command to load template: ```auto filebeat setup --dashboards ``` After that I run this command and output was ok: ```auto filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["X.X.X.X:9200"]' ``` After some changes, Now I get this error when I run again filebeat setup -e: ```auto 2021-04-10T11:03:38.921+0430 WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning. 2021-04-10T11:03:38.921+0430 ERROR instance/beat.go:971 Exiting: Index management requested but the Elasticsearch output is not configured/enabled Exiting: Index management requested but the Elasticsearch output is not configured/enabled ``` I am new. May I ask you help me what should I do step by step to troubleshot or send me some links to study? I would be appreciated you. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 10, 2021, 2:43pm UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/10 "2021-04-10T14:43:15Z")
Hi @fati I provided my suggestion... But looks like you are still trying other options. You are making this more complex than it needs to be. You are trying to run set up with all the individual commands that can often lead to issues as opposed to just running the basic command that I supplied so let's try this again. 1. go into elasticsearch and clean up / delete any filebeat indices 2. in your filebeat.yml configure the output to point to elasticsearch. Comment out the logstash output. The kibana setup looks good. 3. run this command no extra parameters just this command. This will set up everything `filebeat setup -e` If you're running 7. 12 there's actually a little bug and it'll throw some errors at the bottom of that command but it should be fine ignore them for now. 1. now go back into your filebeat.yml and comment out (or take out) the elasticsearch output and put in your logstash output. 2. use the logstash configuration file that I provided above and start logstash. 3. start filebeat with no extra paramaters `filebeat -e` Perhaps I didn't provide enough information there is an nginx pipeline that's automatically loaded and is used to parse the data on the elasticsearch side if you don't configure all this correctly like I showed above that will not be used and therefore your data will not be processed and therefore the dashboards will not work. Please try my suggestions and then if they don't work we can take another look I have set this up many times exactly like this for nginx logs and I can load the dashboards and everything. ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 11, 2021, 3:36am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/11 "2021-04-11T03:36:15Z")
thank you for helping @stephenb. I did exactly 0-5 step by step as you said, but unfortunately the same result. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 11, 2021, 3:51am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/12 "2021-04-11T03:51:04Z")
Hmmmm ... Bummer. ☹ What version Are there any errors? from Filebeat or Logstash? Can you try to send the data directly from filebeat to elasticsearch just to test? (skip logstash) I suspect if you when filebeart -\> Elasticsearch it would work. Can you share what the nginix documents looks like in elasticsearch? `GET filebeat-*/_search` Can you show `GET _cat/indices/filebeat*/?v` Also curious why you have this, are there other logs beside nginx logs? ```auto filebeat.inputs: - type: log enabled: true paths: - /var/log/*.log ``` **Also I just realized somethings I may have found it!!** What other conf files do you have in /etc/logstash/conf.d/ Please do a `ls -l /etc/logstash/conf.d/` And show that results... All conf files in that directory are concatenated together so if you have other conf files they may be interfering, we have see that before.... If you want separate pipelines you define them in the pipelines.yml ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 13, 2021, 3:18am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/13 "2021-04-13T03:18:43Z")
1- I have 5 node ELK stack 7.12.0 (filebeat -\> logstash -\> elasticsearch cluster -\> kibana) elasticsearch = 3 node logstash = 1 node kibana = 1 node 2- I will try to skip logstash and will feedback and I think it can be the problem. 3- GET filebeat-\*/\_search output: woooow, I don't have any ouput for /var/log/nginx as log path. I have only /var/log/messages. 4- GET \_cat/indices/filebeat\*/?v ```auto health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open filebeat-7.12.0-2021.04.11-000001 2KoY1dA6TiqL4ShZoS8GfA 1 1 28928 0 11.1mb 5.5mb ``` ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 13, 2021, 3:23am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/14 "2021-04-13T03:23:57Z")
5- Yes, I gather other logs too. 6- ls -l /etc/logstash/conf.d/ ```auto -rw-r--r-- 1 root root 65 Apr 6 10:49 02-beats-input.conf -rw-r--r-- 1 root root 423 Apr 11 07:31 30-elasticsearch-output.conf ``` There isn't any pipelines.yml file in logstash server. Should I create it? After I change two pipeline to one, the same result again: ls -l /etc/logstash/conf.d/ ```auto -rw-r--r-- 1 root root 488 Apr 13 08:13 sample.conf ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 13, 2021, 4:38am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/15 "2021-04-13T04:38:09Z")
Great info thanks... We will figure this out > [@fati](#): > > 3- GET filebeat-\*/\_search output: > > woooow, I don't have any ouput for /var/log/nginx as log path. I have only /var/log/messages. This is telling.... ^^^^ it looks like filebeat is not harvesting the ngnix logs. On the filebeat server with ngnix logs.... `ls -l /var/log/ngnix` What do you see? I think you maybe missing a wildcard, let's look at that directory. Perhaps it should be `var.paths: ["/var/log/nginx/access.log*"]` Also I would keep it as one Logstash conf file for now, use the ones I gave you... and you don't need a pipeline file for now that's advanced let's figure this out. ---
**Author:** ![fati](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fati](https://discuss.elastic.co/u/fati)\ **Post date:** [April 13, 2021, 6:45am UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/16 "2021-04-13T06:45:38Z")
> [@stephenb](#): > > ls -l /var/log/ngnix ls -l /var/log/nginx/ ```auto -rw-rw-r--. 1 nginx root 18510741 Apr 13 10:30 access.log -rw-rw-r--. 1 nginx root 2472025 Apr 4 03:50 access.log-20210404.gz -rw-rw-r--. 1 nginx root 3482714 Apr 5 03:26 access.log-20210405.gz -rw-rw-r--. 1 nginx root 2884830 Apr 6 03:15 access.log-20210406.gz -rw-rw-r--. 1 nginx root 2589581 Apr 7 03:40 access.log-20210407.gz -rw-rw-r--. 1 nginx root 2420657 Apr 8 03:38 access.log-20210408.gz -rw-rw-r--. 1 nginx root 2153234 Apr 9 03:39 access.log-20210409.gz -rw-rw-r--. 1 nginx root 2347481 Apr 10 03:49 access.log-20210410.gz -rw-rw-r--. 1 nginx root 3130244 Apr 11 03:31 access.log-20210411.gz -rw-rw-r--. 1 nginx root 2950443 Apr 12 03:41 access.log-20210412.gz -rw-rw-r--. 1 nginx root 2875005 Apr 13 03:14 access.log-20210413.gz -rw-rw-r--. 1 nginx root 974454 Apr 13 10:30 error.log -rw-rw-r--. 1 nginx root 131459 Apr 4 03:50 error.log-20210404.gz -rw-rw-r--. 1 nginx root 135832 Apr 5 03:26 error.log-20210405.gz -rw-rw-r--. 1 nginx root 136562 Apr 6 03:15 error.log-20210406.gz -rw-rw-r--. 1 nginx root 139024 Apr 7 03:40 error.log-20210407.gz -rw-rw-r--. 1 nginx root 139061 Apr 8 03:38 error.log-20210408.gz -rw-rw-r--. 1 nginx root 139941 Apr 9 03:39 error.log-20210409.gz -rw-rw-r--. 1 nginx root 140517 Apr 10 03:49 error.log-20210410.gz -rw-rw-r--. 1 nginx root 138689 Apr 11 03:31 error.log-20210411.gz -rw-rw-r--. 1 nginx root 141629 Apr 12 03:41 error.log-20210412.gz -rw-rw-r--. 1 nginx root 138111 Apr 13 03:14 error.log-20210413.gz -rw-r--r--. 1 root root 4495573 May 2 2020 s3.access -rw-r--r--. 1 root root 1653323 May 2 2020 s3.error ``` I just enable filebeat nginx module: ```auto filebeat modules list Enabled: nginx ``` I change path in nginx.yml as you said: ```auto var.paths: ["/var/log/nginx/access.log*"] var.paths: ["/var/log/nginx/error.log*"] ``` in filebeat.yml : ```auto paths: - /var/log/nginx/* ``` ls -l /etc/logstash/conf.d ```auto -rw-r--r-- 1 root root 488 Apr 13 08:13 sample.conf ``` now when I run `GET filebeat-*/_search` the result shows correct paths. but still the same result. I think the problem is filebeat module and it doesn't work. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 13, 2021, 3:15pm UTC](https://discuss.elastic.co/t/nginx-access-and-error-logs-show-incorroct-value-in-kibana-dashboard/269453/17 "2021-04-13T15:15:53Z")
Ok I am willing to keep working if you are... > [@fati](#): > > I think the problem is filebeat module and it doesn't work. Perhaps but this module is used by many people if it wasn't working we would be getting a lot of reports. ... but it may not be working for you, lets figure it out. I think there is 1 of 2 things... 1. The config is not correct 2. Or perhaps your ngnix logs have been modified and they are not standard. **First Important** do not configure the ngnix path in the `filebeat.yml` only the `ngnix.yml` not both. > [@fati](#): > > I change path in nginx.yml as you said: ```auto var.paths: ["/var/log/nginx/access.log*"] var.paths: ["/var/log/nginx/error.log*"] ``` in filebeat.yml : **\