# Unable to get nginx messages to kibana through filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 6, 2017, 7:34pm UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024 "2017-08-06T19:34:52Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 6, 2017, 7:34pm UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/1 "2017-08-06T19:34:52Z")
Hi Guys, I am unable to get the proper messages in elastic through filebeat. I have nginx reverse proxy with at least 50 sites catering to and I just configured or first time gave a try with ELK and filebeat. I am somehow getting messages but those are not proper. Can someone confirm if shipping nginx logs do need any other configuration? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 6, 2017, 11:41pm UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/2 "2017-08-06T23:41:35Z")
> [@Blason](#): > > I am somehow getting messages but those are not proper. What does that mean exactly? ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 7, 2017, 7:09am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/3 "2017-08-07T07:09:08Z")
Well I did receive few messages but those are not indexed and unable to search it. So does filebeat natively supports nginx logs? Or any other parser is needed for the same? Also can I injest DNS logs with filebeat? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 7, 2017, 2:01pm UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/4 "2017-08-07T14:01:11Z")
You have any configs, logs, indexed documents, errors you can share with us? How did you setup/configure the nginx filebeat module? ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 8, 2017, 2:51am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/5 "2017-08-08T02:51:32Z")
Here is the filebeat configuration filebeat: # List of prospectors to fetch data. prospectors: # Each - is a prospector. Below are the prospector specific configurations - # Paths that should be crawled and fetched. Glob based paths. # To fetch all ".log" files from a specific level of subdirectories # /var/log/_/_.log can be used. # For each file found under this path, a harvester is started. # Make sure not file is defined twice as this can lead to unexpected behaviour. paths: - /var/log/nginx/xyz.com/_.log - /var/log/_.log - /var/log/nginx/_/_.log - /var/log/messages - /var/log/secure # Type of the files. Based on this the way the file is read is decided. ``` # The different types cannot be mixed in one prospector # # Possible options are: # * log: Reads every line of the log file (default) # * stdin: Reads the standard in input_type: log ``` Aug 8 08:21:49 labmumwaf01 filebeat: Loading config file error: YAML config parsing failed on /etc/filebeat/filebeat.yml: yaml: line 281: did not find expected key. Exiting. Aug 8 08:21:49 labmumwaf01 systemd: filebeat.service: main process exited, code=exited, status=1/FAILURE Aug 8 08:21:49 labmumwaf01 systemd: Unit filebeat.service entered failed state. Aug 8 08:21:49 labmumwaf01 systemd: filebeat.service holdoff time over, scheduling restart. Aug 8 08:21:49 labmumwaf01 systemd: Stopping filebeat... Aug 8 08:21:49 labmumwaf01 systemd: Starting filebeat... Aug 8 08:21:49 labmumwaf01 systemd: filebeat.service start request repeated too quickly, refusing to start. Aug 8 08:21:49 labmumwaf01 systemd: Failed to start filebeat. Aug 8 08:21:49 labmumwaf01 systemd: Unit filebeat.service entered failed state. ###################### 281 ### Logstash as output 282 logstash: 283 # The Logstash hosts 284 hosts: ["172.16.3.69:5044"] ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 8, 2017, 3:04am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/6 "2017-08-08T03:04:11Z")
And here is the s ![nginxErr](https://us1.discourse-cdn.com/elastic/original/3X/1/1/112beba16fec1cadc411eb1bcc1a4cb8189c9e60.PNG)nap where kibana shows filebeat- index is not found ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 8, 2017, 7:05am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/7 "2017-08-08T07:05:00Z")
> [@Blason](#): > > Aug 8 08:21:49 labmumwaf01 filebeat: Loading config file error: YAML config parsing failed on /etc/filebeat/filebeat.yml: yaml: line 281: did not find expected key. Exiting. You may want to check that line. ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 8, 2017, 8:13am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/8 "2017-08-08T08:13:18Z")
I rectifed that error and it started successfully. However my confusion is since this is an nginx server is it advisable to use logstash parsers through filebeat or can I directly ingest messages from filebeat to elasticsearch? Well I tried pushing messages to elasticsearch directly from filebeat; messages did appear but seems those are not indexed as the entire message appeard in message column and not as source, destination and blah blah. What is most advisable then? Same with sysmon or winlogbeat to elasticsearch? messages do need pushed to logstash or can be directly pushed to elastic through winbeat/sysmon? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 8, 2017, 8:28am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/9 "2017-08-08T08:28:40Z")
> [@Blason](#): > > However my confusion is since this is an nginx server is it advisable to use logstash parsers through filebeat or can I directly ingest messages from filebeat to elasticsearch? Either, there is an nginx module that should make this simpler - [Nginx module | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html) ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 8, 2017, 10:21am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/10 "2017-08-08T10:21:39Z")
That needs to be installed on filebeat server or nginx server I suppose? Please correct me if I am wrong? Or on elasticserach box? ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 8, 2017, 12:08pm UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/11 "2017-08-08T12:08:49Z")
Well I did install ingest-user-agent and ingest-geoip on elk box but not sure where to install the nginx plugin for filebeat? Would you please share the procedure ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 8, 2017, 1:17pm UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/12 "2017-08-08T13:17:15Z")
Oh, you are not using the [nginx filebeat module](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html)? The filebeat module contains kibana dashboards, filebeat configurations and the ingest pipeline configuration for elasticsearch. As you don't use the module, you have to configure the parsing via Elasticsearch ingest pipeline or logstash yourself. See the [nginx module sources](https://github.com/elastic/beats/tree/master/filebeat/module/nginx), for Ingest Node configs (`/ingest/default.json` files). The `/config/...` files do include templates for building the filebeat prospector settings. The `nginx/_meta` directory contains the kibana dashboards. Check out the [Module Overview] and [Tutorial] docs, to get started with modules. It's much more user-friendly, then having to configure everything yourself. You prospector also includes all the different logs. Consider defining multiple prospectors per log-type. This allows you to add additional meta-data to the different log types and configure another processing pipeline in ES Ingest Node or Logstash. This is how modules work, they create specialized prospector configurations in filebeat. ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 8, 2017, 5:37pm UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/13 "2017-08-08T17:37:28Z")
I just somehow figured out the modules and ran on filebeat machine however I am getting below error on kibana dashboard and unable to see the dashboards. Can someone pls help? Error Saved Visualization Service: Visualization type of "tagcloud" is invalid. Please change to a valid type. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 9, 2017, 11:08am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/14 "2017-08-09T11:08:03Z")
Sounds like the dashboard is not fully compatible with your kibana version. ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [August 9, 2017, 11:48am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/15 "2017-08-09T11:48:47Z")
I am using 5.x do I need to use anything else. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 6, 2017, 11:49am UTC](https://discuss.elastic.co/t/unable-to-get-nginx-messages-to-kibana-through-filebeat/96024/16 "2017-09-06T11:49:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.