# How do i make sure that logstash received data from filebeat? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 10, 2016, 2:08pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666 "2016-05-10T14:08:14Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 10, 2016, 2:08pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/1 "2016-05-10T14:08:14Z")
I am trying to use the filebeat and ELK for the first time. I have my components running on RHEL5 server. Filebeat is running on another server with RHEL6 as OS and ELK is running in a 3rd server with RHEL6 as OS. As suggested by many experts, i have mounted the logs folder from the RHEL5 machine to filebeat server. So now filebeat server can access the logs folder. Following is the filebeat configuration file: ```auto ############################# Filebeat ###################################### filebeat: # List of prospectors to fetch data. prospectors: # Each - is a prospector. Below are the prospector specific configurations - # Paths that should be crawled and fetched. Glob based paths. # To fetch all ".log" files from a specific level of subdirectories # /var/log/*/*.log can be used. # For each file found under this path, a harvester is started. # Make sure not file is defined twice as this can lead to unexpected behaviour. paths: - /mnt/cmdc_logs/*.audit logstash: # The Logstash hosts hosts: ["10.209.26.151:5044"] ``` How can i verify that filebeat is sending all the logs from the mounted folder to logstash? Is there any logs? I dont see anything in Kibana. Once i resolve this, i can check why Kibana is not showing anything. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 10, 2016, 2:53pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/2 "2016-05-10T14:53:41Z")
Here you can find the different logging options: [https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-logging.html](https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-logging.html) Be aware that we do not recommend to use mounted volumes: [https://www.elastic.co/guide/en/beats/filebeat/1.2/filebeat-network-volumes.html](https://www.elastic.co/guide/en/beats/filebeat/1.2/filebeat-network-volumes.html) ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 12, 2016, 10:53am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/3 "2016-05-12T10:53:17Z")
I am getting this error in the log. 2016-05-12T10:51:43Z CRIT Unable to publish events to console: write /dev/stdout: invalid argument 2016-05-12T10:51:43Z ERR Error sending/writing event: write /dev/stdout: invalid argument Any idea why is this? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 12, 2016, 12:31pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/4 "2016-05-12T12:31:23Z")
/dev/stdout? You changed your config? ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 12, 2016, 12:33pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/5 "2016-05-12T12:33:43Z")
This issue is seen on filebeat. I didnt change the config at all..Can i attach my config for your reference? ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 13, 2016, 10:18am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/6 "2016-05-13T10:18:17Z")
Steffens Can you please help me in this? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 13, 2016, 1:23pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/7 "2016-05-13T13:23:35Z")
Please attach your full config file. ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 16, 2016, 8:26am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/8 "2016-05-16T08:26:06Z")
Steffens... Please find the configuration file used for logstash. input { beats { port=\>5044 } } output { elasticsearch { hosts =\> ["10.209.26.147:9200"] manage\_template =\> false } } I am afraid that i dont see any other config files used for logs. I dont see a logs folder also. ---
**Author:** ![karuneshupadhyay](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@karuneshupadhyay](https://discuss.elastic.co/u/karuneshupadhyay)\ **Post date:** [May 17, 2016, 6:03am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/9 "2016-05-17T06:03:38Z")
Hi Roshan, Where did you add this piece of code - logging: level: warning # enable file rotation with default configuration to\_files: true # do not log to syslog to\_syslog: false files: path: /var/log/mybeat name: mybeat.log keepfiles: 7 is it in filebeat.yml ? Thanks and Regards, Karunesh ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 17, 2016, 8:10am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/10 "2016-05-17T08:10:40Z")
Hello, This is in filebeat.yml ---
**Author:** ![karuneshupadhyay](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@karuneshupadhyay](https://discuss.elastic.co/u/karuneshupadhyay)\ **Post date:** [May 17, 2016, 9:46am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/11 "2016-05-17T09:46:46Z")
Hi Roshan, Are you able to see the output of filebeat? I am also facing same problem , I also want to know the file transfer status and details about it. I did some configuration , but not able to see any log files . Thanks and Regards, Karunesh Upadhyay ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 17, 2016, 3:05pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/12 "2016-05-17T15:05:53Z")
Can you please share your full filebeat config file? ---
**Author:** ![devendra.address](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devendra.address/32/9652_2.png) [@devendra.address](https://discuss.elastic.co/u/devendra.address)\ **Post date:** [May 17, 2016, 5:18pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/13 "2016-05-17T17:18:36Z")
First need to mapping for filebeat index in kibana, default index pattern is filebeat-\* ---
**Author:** ![devendra.address](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devendra.address/32/9652_2.png) [@devendra.address](https://discuss.elastic.co/u/devendra.address)\ **Post date:** [May 17, 2016, 5:23pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/14 "2016-05-17T17:23:33Z")
```auto filebeat: prospectors: - paths: - /var/log/secure - /var/log/messages # - /var/log/*.log input_type: log document_type: syslog registry_file: /var/lib/filebeat/registry output: logstash: hosts: ["123.12.3.54:5044"] bulk_max_size: 1024 tls: certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"] shipper: name: XYZ logging: files: rotateeverybytes: 10485760 # = 10MB ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 17, 2016, 7:47pm UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/15 "2016-05-17T19:47:48Z")
Too many unrelated configs? Sorry, I totally lost track. @Roshan_r can you please post your filebeat config file? You get same error when running filebeat on console? `$ filebeat -e -v -c ` ? ---
**Author:** ![karuneshupadhyay](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@karuneshupadhyay](https://discuss.elastic.co/u/karuneshupadhyay)\ **Post date:** [May 18, 2016, 6:13am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/16 "2016-05-18T06:13:04Z")
Hi Ruflin, I am new to filebeat and logstash. This is my first time. ################## Filebeat Configuration Example ######################### ############################# Filebeat ###################################### filebeat: prospectors: ``` paths: - /opt/apache-tomcat-7.0.69/logs/* input_type: log ``` logstash: ``` hosts: ["localhost:5044"] ``` logging: to\_syslog: false to\_files: true files: ``` path: "/var/log" ``` name: filebeat.log rotateeverybytes: 10485760 # = 10MB keepfiles: 7 selectors: ["\*"] level: warning #########################################################33 Output is Logstash . starting Filebeat - ./filebeat -e -v -d '\*' . It is running but not able to log folder. Thanks and Regards, Karunesh Upadhyay ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 18, 2016, 7:49am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/17 "2016-05-18T07:49:31Z")
steffens, The full config is as below for filebeat. ``` filebeat: prospectors: - paths: - /mnt/cmdc_logs/*.audit*" input_type: log document_type: my_log output: logstash: hosts: ["10.209.26.147:5044"] console: pretty: true shipper: logging: to_files: true files: path: /var/log/mybeat # The name of the files where the logs are written to. name: mybeat # Configure log file size limit. If limit is reached, log file will be # automatically rotated rotateeverybytes: 10485760 # = 10MB # Number of rotated log files to keep. Oldest files will be deleted first. keepfiles: 7 #selectors: [] level: info ``` Please find the output when i run. I dont see any errors here. [root@astroHeka filebeat]# service filebeat start Starting filebeat: 2016/05/18 07:49:18.047889 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths 2016/05/18 07:49:18.047934 outputs.go:126: INFO Activated console as output plugin. 2016/05/18 07:49:18.048154 logstash.go:106: INFO Max Retries set to: 3 2016/05/18 07:49:18.051179 outputs.go:126: INFO Activated logstash as output plugin. 2016/05/18 07:49:18.051793 publish.go:288: INFO Publisher name: astroHeka 2016/05/18 07:49:18.058809 async.go:78: INFO Flush Interval set to: 1s 2016/05/18 07:49:18.058832 async.go:84: INFO Max Bulk Size set to: 2048 2016/05/18 07:49:18.058931 async.go:78: INFO Flush Interval set to: 1s 2016/05/18 07:49:18.058947 async.go:84: INFO Max Bulk Size set to: 2048 2016/05/18 07:49:18.058998 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.2 [OK] ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 18, 2016, 9:51am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/18 "2016-05-18T09:51:58Z")
devendra... where i should insert the default index pattern? I am getting an error in elasticsearch saying an issue with default index ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 18, 2016, 11:10am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/19 "2016-05-18T11:10:09Z")
The indentation looks weird. I can not tell if due to copy'n paste or indentation is off for real. beats use YAML format which is very sensitive to indentation. > [@Roshan\_r](#): > > 2016/05/18 07:49:18.047934 outputs.go:126: INFO Activated console as output plugin. This could be your problem. When running filebeat as service, stdout might be closed. Comment out `output.console` section in config file. ---
**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Post date:** [May 18, 2016, 11:12am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666/20 "2016-05-18T11:12:19Z")
The indentation went wrong when i copy and pasted. What does it say from the logs when i start the service as it does not show any errors. [Next page](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-filebeat/49666.md?page=2)