File beat configuration to ship CSV file to logstash

theiamdude · August 4, 2016, 4:50am

Hi all,
I am trying to ship CSV file to my logstash instance and every time I start the process it get's stuck at 2016/08/04 04:38:18.693034 log.go:113: INFO Harvester started for file: ../logs/ldap-access.csv

It works file if I just send a nomal log file, but for some reason the CSV file does not ship with no obvious errors.

Here is my filebeat configuration:

gist.github.com

https://gist.github.com/theiamdude/5825885a853037ba6e30c907ca893635

filebeat_csv.yml

filebeat:
  prospectors:
      paths:
        #- /var/log/*.log
        - "../logs/ldap*.csv"
        #- ../logs/access ## Wokring with no issue
        #- c:\programdata\elasticsearch\logs\*
      input_type: log
      document_type: ldap_access
  logstash:

This file has been truncated. show original

The logs I am getting are:

gist.github.com

https://gist.github.com/theiamdude/4978a61a73babd5a32ce417856d17ab0

filebeat.log

[root@ip-0-0-0-0 filebeat-1.2.3-x86_64]# ./filebeat -e -d "publish"
2016/08/04 04:38:18.690593 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/08/04 04:38:18.690750 logstash.go:106: INFO Max Retries set to: 3
2016/08/04 04:38:18.691240 outputs.go:126: INFO Activated logstash as output plugin.
2016/08/04 04:38:18.691253 publish.go:232: DBG  Create output worker
2016/08/04 04:38:18.691310 publish.go:274: DBG  No output is defined to store the topology. The server fields might not be filled.
2016/08/04 04:38:18.691341 publish.go:288: INFO Publisher name: ip-0-0-0-0.ap-southeast-2.compute.internal
2016/08/04 04:38:18.691791 async.go:78: INFO Flush Interval set to: 1s
2016/08/04 04:38:18.691801 async.go:84: INFO Max Bulk Size set to: 2048
2016/08/04 04:38:18.691810 async.go:92: DBG  create bulk processing worker (interval=1s, bulk size=2048)

This file has been truncated. show original

Any help will be greatly appreciated.

Cheers,
M.C

ruflin · August 8, 2016, 8:09am

Can you share some content of your csv log file? does it have newlines inside? As you have debug enabled, I would expect some more output after a few seconds.

system · August 25, 2016, 4:50am

This topic was automatically closed after 21 days. New replies are no longer allowed.