Filebeat loses connection to Kafka


#1

I am using the latest nightly build of Filebeat to output logs from /var/log/messages to Kafka (single node).

The whole thing is working fine for about 30mins and after that Filebeat suddenly stops being able to communicate with Kafka.

Restarting the filebeat service then fixes the issue.

Below the logs I'm getting at the moment when the problem occurs:

2016-04-18T13:20:14+03:00 DBG  output worker: publish 2 events
2016-04-18T13:20:14+03:00 DBG  guaranteed flag is set
2016-04-18T13:20:14+03:00 DBG  publish events with attempts=-1
2016-04-18T13:20:14+03:00 DBG  forwards msg with attempts=-1
2016-04-18T13:20:14+03:00 DBG  message forwarded
2016-04-18T13:20:14+03:00 DBG  events from worker worker queue
2016-04-18T13:20:14+03:00 DBG  publish events
2016-04-18T13:20:14+03:00 WARN producer/broker/[1 859530387616] state change to [closing] because %!s(MISSING)

2016-04-18T13:20:14+03:00 WARN Closed connection to broker [172.24.33.7:9092]

2016-04-18T13:20:14+03:00 DBG  Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG  Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG  finished kafka batch
2016-04-18T13:20:14+03:00 DBG  handlePublishEventsResult
2016-04-18T13:20:14+03:00 DBG  handle publish error: EOF
2016-04-18T13:20:14+03:00 INFO Error publishing events (retrying): EOF

After that I'm getting infinite number of repeated logs looking like that:

2016-04-18T13:20:14+03:00 DBG  forwards msg with attempts=-1
2016-04-18T13:20:14+03:00 DBG  message forwarded
2016-04-18T13:20:14+03:00 DBG  events from retries queue
2016-04-18T13:20:14+03:00 DBG  publish events
2016-04-18T13:20:14+03:00 DBG  Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG  Kafka publish failed with: EOF
2016-04-18T13:20:14+03:00 DBG  finished kafka batch
2016-04-18T13:20:14+03:00 DBG  handlePublishEventsResult
2016-04-18T13:20:14+03:00 DBG  handle publish error: EOF
2016-04-18T13:20:14+03:00 INFO Error publishing events (retrying): EOF

Could that be a bug in filebeat?


(Steffen Siering) #2

looks like a bug kafka client not reconnecting.


#3

Should I therefore create an issue here: https://github.com/elastic/beats/issues ?


(Steffen Siering) #4

Yes please, I didn't find a ticket. Thanks.


(system) #5