Filebeat dies after trying to harvest 3 times (max_retries)

Hi,

I have some problems when getting logs after log file haven't changed after 3 retries of harvesting, filebeat dies after this retries and I have to start manually to start harvesting.

This is the log in Filebeat before it dies (this fragment appears three times in total):

2019-04-18T17:09:54.324-0700    DEBUG   [input] input/input.go:152      Run input
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:174        Start next scan
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:404        Check file for harvesting: /app/logs/A/api.log
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:494        Update existing file for harvesting: /app/logs/A/api.log, offset: 2922198
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:548        File didn't change: /app/logs/A/api.log

2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:404        Check file for harvesting: /app/logs/B/api.log
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:494        Update existing file for harvesting: /app/logs/B/api.log, offset: 28149
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:548        File didn't change: /app/logs/B/api.log

2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:404        Check file for harvesting: /app/logs/C/api.log
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:494        Update existing file for harvesting: /app/logs/C/api.log, offset: 37930655
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:546        Harvester for file is still running: /app/logs/C/api.log

2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:404        Check file for harvesting: /app/logs/D/api.log
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:494        Update existing file for harvesting: /app/logs/D/api.log, offset: 173342
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:548        File didn't change: /app/logs/D/api.log
2019-04-18T17:09:54.324-0700    DEBUG   [input] log/input.go:195        input states cleaned up. Before: 4, After: 4, Pending: 0

In filebeat.yml this is the configuration for process in port 5044. I changed in filebeat.yml max_retries:3 to -1 (infinite) as follows:

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["log1.cgi-dev.ca:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: 

  # Certificate for SSL client authentication
  #ssl.certificate: 

  # Client Certificate Key
  #ssl.key: 

  # The number of times to retry publishing an event after a publishing failure.
  # After the specified number of retries, the events are typically dropped.
  # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting
  # and retry until all events are published.  Set max_retries to a value less
  # than 0 to retry until all events are published. The default is 3.
  max_retries: -1

I am getting successfully the logs from filebeat, but when the log file does not change for a time, the filebeat process dies. What would be the culprit to cause Filebeat to die? What would be the configuration might cause this problem? Should I create a daemon to ensure the filebeat process in up and start it if the process goes down?

Thanks

I'm using a machine builded by bitnami. When I access the dashboard Kibana by URL, show me the things are working good but if I try output test (filebeat test output) the conection was refused. I'm using default port filebeat (5044) and it is open on machine.

I'm trying figure out how to stay the conection open full time.

How did you start filebeat? It's supposed to be a daemon and shouldn't just stop. If the connection is lost, filebeat just tries to reconnect until it found a stable connection. But still it should not just die in this case.

I just run:

sudo filebeat

Now I'm confused. @TiNhO how is this related to the original issue by Arthur? The filebeat test output is a one time action. If the remote is not available you will get a connection refused (e.g. if Logstash is not running), still filebeat can start and tries to reconnect. Please consider to open a separate issue

@steffens NVM. I found out that my logstash is dying from some reason. For that reason filebeat show me connection refused. Tnks 4 ur help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.