Sending batch to Kafka failed retry loop error

[jamiebuxxx (Ja Mile Jackson)]

Hello,

I recently updated our log-brokers to use Logstash 5.6.11. This version is currently using the logstash-output-kafka plugin version 5.1.11. We have 3 log brokers and we recently noticed connections from Kafka dropping from 2/3 of the brokers. We see the following messages in the logs.

tail -f /var/log/logstash-broker/logstash-plain.log
[2018-09-17T16:59:53,522][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:51400"}
[2018-09-17T16:59:53,520][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:51401"}
[2018-09-17T16:59:53,529][INFO ][logstash.pipeline        ] Pipeline main started
[2018-09-17T16:59:53,536][INFO ][logstash.inputs.gelf     ] Starting gelf listener (udp) ... {:address=>"0.0.0.0:12201"}
[2018-09-17T16:59:53,547][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:51401", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2018-09-17T16:59:53,550][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:51400", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2018-09-17T16:59:53,551][INFO ][logstash.inputs.gelf     ] Starting gelf listener (udp) ... {:address=>"0.0.0.0:12202"}
[2018-09-17T16:59:53,586][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-09-17T16:59:53,575][INFO ][org.logstash.beats.Server] Starting server on port: 5045
[2018-09-17T16:59:53,614][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-09-17T17:35:58,665][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}
[2018-09-17T17:35:58,679][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}
[2018-09-17T17:35:58,692][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}

The last message just loops until the either the Logstash service is restarted. We have changed nothing from our Kafka cluster or Logstash configs, only the Logstash version.

logstash.conf:

...
output{
  kafka {
    topic_id => "logstash-broker"
    bootstrap_servers => "logging-kafka-02.domain:9092"
    codec => json
    id => "logging_kafkfa_broker"
  }
}

I saw a couple of related issues on the forum which mentioned to make sure the Kafka server.properties listeners to PLAINTEXT. We currently have that set already.

kafka server.properties

listeners=PLAINTEXT://0.0.0.0:9092

In the meantime, we are going to have to revert back to the older version of Logstash(5.2) until we are able to resolve this issue. Any help or direction is much appreciated.