Ran into a weird issue. I have Filebeat picking up logs on a centos7 vm and sending them over to instances of Graylog using the Logstash output.
I having Metricbeat and Packetbeat saving to disk, so Filebeat would pick up their logs along with the rest of the linux logs.
Filebeat will work for awhile and then just stop forwarding any logs. If I restart Filebeat, it will then send again picking up where it left off, until it just stops again. The service still appears to be running and there is no indication of an error in the filebeat log file. Instead I get just this repeating:
2017-04-18T19:05:20Z INFO No non-zero metrics in the last 30s
I'm running Filebeat version 5.1.2.
Here is my Prospector configuration:
- input_type: log
paths:- /var/log/*
- /var/log//.log
exclude_files: ['.gz$', '.bz2$', '.zip$', '.tar$', 'filebeat']
ignore_older: 96h
I was trying out different backoff, close, clean options but commented them all out to rule out an error from these and just use defaults.
The /var/lib/filebeat/registry timestamp stops at the same time. It is only 7.4K in size, I count only 50 sources being tracked in it.
In case this is helpful, metric and packet logs are only 10M before rotating, and include_fields are keeping the json output a lot leaner than default.
As a test, I shutoff metric and packetbeat. Restarted filebeat. Waited until it kept repeating the non-zero metrics message. Started Metricbeat. In the filebeat log I then see "Non-zero metrics in the last 30s: filebeat.harvester.open_files=1". Then the next messages again are No non-zero metrics, and I don't get anything on the Graylog side.
In my logstash output, I have 2 hosts, loadbalance is true, worker is 2, pipelining is 2, timeout is 10, max_retries is 65500.
If it was a connectivity issue, wouldn't I be seeing a recurring error about it, especially with the high max retries?
Thanks in advance for any suggestions -