Filebeat not sending logs to Elasticsearch

Elastic Stack Beats
1

Hi,
I am new to Filebeat and cannot get the logs to reach Elasticsearch. I have Filebeat and Elasticsearch on the same Ubuntu instance.

Below is a snapshot from the filebeat logs:

2016-06-15T14:38:00-07:00 DBG Disable stderr logging
2016-06-15T14:38:00-07:00 INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2016-06-15T14:38:00-07:00 DBG filters:
2016-06-15T14:38:00-07:00 INFO Setup Beat: filebeat; Version: 5.0.0-alpha3
2016-06-15T14:38:00-07:00 DBG Initializing output plugins
2016-06-15T14:38:00-07:00 INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2016-06-15T14:38:00-07:00 INFO Elasticsearch url: http://localhost:9200
2016-06-15T14:38:00-07:00 INFO Activated elasticsearch as output plugin.
2016-06-15T14:38:00-07:00 DBG Create output worker
2016-06-15T14:38:00-07:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-06-15T14:38:00-07:00 INFO Publisher name: ubuntu
2016-06-15T14:38:00-07:00 INFO Flush Interval set to: 1s
2016-06-15T14:38:00-07:00 INFO Max Bulk Size set to: 50
2016-06-15T14:38:00-07:00 DBG create bulk processing worker (interval=1s, bulk size=50)
2016-06-15T14:38:00-07:00 INFO filebeat start running.
2016-06-15T14:38:00-07:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-06-15T14:38:00-07:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-06-15T14:38:00-07:00 DBG Error decoding old state: json: cannot unmarshal array into Go value of type map[string]input.FileState
2016-06-15T14:38:00-07:00 INFO States Loaded from registrar: 3
2016-06-15T14:38:00-07:00 DBG Spooler will use the default spool_size of 2048
2016-06-15T14:38:00-07:00 DBG Spooler will use the default idle_timeout of 5s
2016-06-15T14:38:00-07:00 INFO Loading Prospectors: 1
2016-06-15T14:38:00-07:00 INFO buffer_size set to: 16384
2016-06-15T14:38:00-07:00 INFO document_type set to: log
2016-06-15T14:38:00-07:00 INFO input_type set to: log
2016-06-15T14:38:00-07:00 INFO Set backoff duration to 1s
2016-06-15T14:38:00-07:00 INFO backoff_factor set to: 2
2016-06-15T14:38:00-07:00 INFO Set max_backoff duration to 10s
2016-06-15T14:38:00-07:00 INFO force_close_file is disabled
2016-06-15T14:38:01-07:00 INFO Set close_older duration to 1h0m0s
2016-06-15T14:38:01-07:00 INFO max_bytes set to: 10485760
2016-06-15T14:38:01-07:00 DBG exclude_files: []
2016-06-15T14:38:01-07:00 INFO Load previous states from registry into memory
2016-06-15T14:38:01-07:00 INFO Previous states loaded: 3
2016-06-15T14:38:01-07:00 DBG File Configs: [/home/kshah/Downloads/logs/log3.log]
2016-06-15T14:38:01-07:00 INFO Loading Prospectors completed. Number of prospectors: 1
2016-06-15T14:38:01-07:00 INFO All prospectors are initialised and running with 3 states to persist
2016-06-15T14:38:01-07:00 INFO Start sending events to output
2016-06-15T14:38:01-07:00 INFO Starting Registrar
2016-06-15T14:38:01-07:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016-06-15T14:38:01-07:00 DBG Starting prospector 0
2016-06-15T14:38:01-07:00 INFO Starting prospector of type: log
2016-06-15T14:38:01-07:00 DBG Start next scan
2016-06-15T14:38:01-07:00 DBG Check file for harvesting: /home/kshah/Downloads/logs/log3.log
2016-06-15T14:38:01-07:00 DBG Update existing file for harvesting: /home/kshah/Downloads/logs/log3.log, offset: 446
2016-06-15T14:38:01-07:00 DBG Resuming harvesting of file: /home/kshah/Downloads/logs/log3.log, offset: 446
2016-06-15T14:38:01-07:00 DBG harvest: "/home/kshah/Downloads/logs/log3.log" position:446 (offset snapshot:0)
2016-06-15T14:38:01-07:00 INFO Harvester started for file: /home/kshah/Downloads/logs/log3.log
2016-06-15T14:38:01-07:00 DBG Update state: /home/kshah/Downloads/logs/log3.log, offset: 446
2016-06-15T14:38:01-07:00 DBG Old state overwritten for /home/kshah/Downloads/logs/log3.log
2016-06-15T14:38:01-07:00 DBG End of file reached: /home/kshah/Downloads/logs/log3.log; Backoff now.
2016-06-15T14:38:02-07:00 DBG End of file reached: /home/kshah/Downloads/logs/log3.log; Backoff now.
2016-06-15T14:38:04-07:00 DBG End of file reached: /home/kshah/Downloads/logs/log3.log; Backoff now.
2016-06-15T14:38:06-07:00 DBG Flushing spooler because of timeout. Events flushed: 1
2016-06-15T14:38:06-07:00 DBG No events to publish
2016-06-15T14:38:06-07:00 INFO Events sent: 1
2016-06-15T14:38:06-07:00 DBG Processing 1 events
2016-06-15T14:38:06-07:00 DBG Old state overwritten for /home/kshah/Downloads/logs/log3.log
2016-06-15T14:38:06-07:00 DBG Write registry file: /var/lib/filebeat/registry
2016-06-15T14:38:06-07:00 INFO Registry file updated. 3 states written.
2016-06-15T14:38:08-07:00 DBG End of file reached: /home/kshah/Downloads/logs/log3.log; Backoff now.

Has anyone faced this issue? Any help is appreciated.

Thanks

2

It seems like no events are sent, as filebeat continues at a previous position. I assume you started filebeat before. Make sure to stop filebeat, remove the registry file and start it again so it starts sending from scratch.

3

Thanks. That worked.
Do I need to remove it everytime? what is the significance of registry file?

4

The registry file controls where the process reads from each time it opens the file.
You will only need to do this if you want to reprocess the file.

If you are still testing, then try setting the file to /dev/null, then it'll never create one.

5

Ok. Thanks.

6

This topic was automatically closed after 21 days. New replies are no longer allowed.