File beat configuration to ship CSV file to logstash

theiamdude · August 4, 2016, 4:50am

Hi all,
I am trying to ship CSV file to my logstash instance and every time I start the process it get's stuck at 2016/08/04 04:38:18.693034 log.go:113: INFO Harvester started for file: ../logs/ldap-access.csv

It works file if I just send a nomal log file, but for some reason the CSV file does not ship with no obvious errors.

Here is my filebeat configuration:

gist.github.com

https://gist.github.com/theiamdude/5825885a853037ba6e30c907ca893635

filebeat_csv.yml

filebeat:
  prospectors:
      paths:
        #- /var/log/*.log
        - "../logs/ldap*.csv"
        #- ../logs/access ## Wokring with no issue
        #- c:\programdata\elasticsearch\logs\*
      input_type: log
      document_type: ldap_access
  logstash:

This file has been truncated. show original

The logs I am getting are:

gist.github.com

https://gist.github.com/theiamdude/4978a61a73babd5a32ce417856d17ab0

filebeat.log

[root@ip-0-0-0-0 filebeat-1.2.3-x86_64]# ./filebeat -e -d "publish"
2016/08/04 04:38:18.690593 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/08/04 04:38:18.690750 logstash.go:106: INFO Max Retries set to: 3
2016/08/04 04:38:18.691240 outputs.go:126: INFO Activated logstash as output plugin.
2016/08/04 04:38:18.691253 publish.go:232: DBG  Create output worker
2016/08/04 04:38:18.691310 publish.go:274: DBG  No output is defined to store the topology. The server fields might not be filled.
2016/08/04 04:38:18.691341 publish.go:288: INFO Publisher name: ip-0-0-0-0.ap-southeast-2.compute.internal
2016/08/04 04:38:18.691791 async.go:78: INFO Flush Interval set to: 1s
2016/08/04 04:38:18.691801 async.go:84: INFO Max Bulk Size set to: 2048
2016/08/04 04:38:18.691810 async.go:92: DBG  create bulk processing worker (interval=1s, bulk size=2048)

This file has been truncated. show original

Any help will be greatly appreciated.

Cheers,
M.C

ruflin · August 8, 2016, 8:09am

Can you share some content of your csv log file? does it have newlines inside? As you have debug enabled, I would expect some more output after a few seconds.

system · August 25, 2016, 4:50am

This topic was automatically closed after 21 days. New replies are no longer allowed.

Topic		Replies	Views
Shipping CSV file using Filebeat to Logstash and then from logstash to Elasticsearch Logstash	3	8182	July 6, 2017
How to send CSV from Filebeat to Logstash Beats filebeat	10	6715	May 22, 2017
How to send csv file to logstash using filebeat Logstash	5	3917	June 19, 2018
Configuring filebeat to transport csv Logstash	3	904	December 12, 2019
Csv -> filebeat -> logstash not parsing? Beats filebeat	2	1081	July 1, 2018

File beat configuration to ship CSV file to logstash

Related topics