Hi,
I have some problems when getting logs after log file haven't changed after 3 retries of harvesting, filebeat dies after this retries and I have to start manually to start harvesting.
This is the log in Filebeat before it dies (this fragment appears three times in total):
2019-04-18T17:09:54.324-0700 DEBUG [input] input/input.go:152 Run input
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:174 Start next scan
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/A/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/A/api.log, offset: 2922198
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:548 File didn't change: /app/logs/A/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/B/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/B/api.log, offset: 28149
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:548 File didn't change: /app/logs/B/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/C/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/C/api.log, offset: 37930655
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:546 Harvester for file is still running: /app/logs/C/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/D/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/D/api.log, offset: 173342
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:548 File didn't change: /app/logs/D/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:195 input states cleaned up. Before: 4, After: 4, Pending: 0
In filebeat.yml this is the configuration for process in port 5044. I changed in filebeat.yml max_retries:3 to -1 (infinite) as follows:
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["log1.cgi-dev.ca:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities:
# Certificate for SSL client authentication
#ssl.certificate:
# Client Certificate Key
#ssl.key:
# The number of times to retry publishing an event after a publishing failure.
# After the specified number of retries, the events are typically dropped.
# Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting
# and retry until all events are published. Set max_retries to a value less
# than 0 to retry until all events are published. The default is 3.
max_retries: -1
I am getting successfully the logs from filebeat, but when the log file does not change for a time, the filebeat process dies. What would be the culprit to cause Filebeat to die? What would be the configuration might cause this problem? Should I create a daemon to ensure the filebeat process in up and start it if the process goes down?
Thanks