LS output failed

Elastic Stack Logstash
1

I'm seeing the following INFO message, the connection to Kafka is lost and I'm unable to connect to it.

2018-03-14T14:13:02,743][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}

I've validate that Kafka is up and running.

listeners=PLAINTEXT://IP:9092

this line already exist in Kafka. I could telnet to the Kafka server even though LS is unable to output to it.

ES 5.6.8
KF 0.10.2.0

2

any help would be great?

3

ran a debug, but got way too many logs. I noticed that it failed at exactly 19:00 three days in a row.

4

Does the connection ever resume? Does it require a restart of Logstash?

The timing is suspicious; could a scheduled task be restarting the Kafka process? In a previous job, we had a finnicky Kafka installation that someone "fixed" by killing it daily in cron.

5

It doesn’t reconnect. I have to restart Logstash to get the connection going.

6

I found something interesting.

If I reboot the box, it will work, but a few hours later, the error is back. But if I restart it using "initctl" it will work fine for a while. What the different in the two starting method?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.