I am using the latest nightly build of Filebeat to output logs from /var/log/messages to Kafka (single node).
The whole thing is working fine for about 30mins and after that Filebeat suddenly stops being able to communicate with Kafka.
Restarting the filebeat service then fixes the issue.
Below the logs I'm getting at the moment when the problem occurs:
2016-04-18T13:20:14+03:00 DBG output worker: publish 2 events
2016-04-18T13:20:14+03:00 DBG guaranteed flag is set
2016-04-18T13:20:14+03:00 DBG publish events with attempts=-1
2016-04-18T13:20:14+03:00 DBG forwards msg with attempts=-1
2016-04-18T13:20:14+03:00 DBG message forwarded
2016-04-18T13:20:14+03:00 DBG events from worker worker queue
2016-04-18T13:20:14+03:00 DBG publish events
2016-04-18T13:20:14+03:00 WARN producer/broker/[1 859530387616] state change to [closing] because %!s(MISSING)
2016-04-18T13:20:14+03:00 WARN Closed connection to broker [172.24.33.7:9092]
2016-04-18T13:20:14+03:00 DBG Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG finished kafka batch
2016-04-18T13:20:14+03:00 DBG handlePublishEventsResult
2016-04-18T13:20:14+03:00 DBG handle publish error: EOF
2016-04-18T13:20:14+03:00 INFO Error publishing events (retrying): EOF
After that I'm getting infinite number of repeated logs looking like that:
2016-04-18T13:20:14+03:00 DBG forwards msg with attempts=-1
2016-04-18T13:20:14+03:00 DBG message forwarded
2016-04-18T13:20:14+03:00 DBG events from retries queue
2016-04-18T13:20:14+03:00 DBG publish events
2016-04-18T13:20:14+03:00 DBG Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG finished kafka batch
2016-04-18T13:20:14+03:00 DBG handlePublishEventsResult
2016-04-18T13:20:14+03:00 DBG handle publish error: EOF
2016-04-18T13:20:14+03:00 INFO Error publishing events (retrying): EOF
Could that be a bug in filebeat?