Logstash keeps crashing

[crapoulou]

Unfortunately, today, I get the same issue.
Finally, a wrong fix found regarding the max size.

It looks the same issue than these topics but my config is correct in /opt/kafka/config/server.properties !
listeners=PLAINTEXT://1.2.3.4:9092 #(IP changed)

Sending batch to Kafka failed. Will retry after a delay Logstash error

Hi, I have a Logstash agent sending lines to a multi broker kafka setup. Currently I am getting the following error in Logstash log: [2017-12-27T15:06:15,369][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>125, :failures=>125, :sleep=>0.01} [2017-12-27T15:06:15,370][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>125, :failures=>125, :sleep=>0.01} [2017-12-27T15:06:15,370][INFO…

Logstash output plugin kafka is not pushing data into kafka

After updating all logstash plugins now i am seeing following logs [2017-10-18T19:55:36,794][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>6, :failures=>6, :sleep=>0.01} [2017-10-18T19:56:06,901][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01} [2017-10-18T19:56:06,903][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry aft…

(Indeed, I see in the start sequence these two parameters) :
security.protocol = PLAINTEXT
bootstrap.servers = [1.2.3.4:9092, 1.2.3.5:9092]

It's not feeding through kafka.

In debug mode I find :
[2018-01-12T11:49:10,544][DEBUG][org.apache.kafka.clients.NetworkClient] Initiating connection to node -1 at 1.2.3.4:9092.
[2018-01-12T11:49:10,546][DEBUG][org.apache.kafka.common.metrics.Metrics] Added sensor with name node--1.bytes-sent
[2018-01-12T11:49:10,547][DEBUG][org.apache.kafka.common.metrics.Metrics] Added sensor with name node--1.bytes-received
[2018-01-12T11:49:10,547][DEBUG][org.apache.kafka.common.metrics.Metrics] Added sensor with name node--1.latency
[2018-01-12T11:49:10,547][DEBUG][org.apache.kafka.common.network.Selector] Connection with /1.2.3.4 disconnected
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_151]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_151]
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:51) ~[kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:73) ~[kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:309) [kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.common.network.Selector.poll(Selector.java:283) [kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260) [kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229) [kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134) [kafka-clients-0.10.0.1.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

I tried to shutdown one of the servers. I have only 1 error message from time to time "Sending batch to Kafka failed. Will retry after a delay."

Once started, I have many times this error !

Do you have any idea ??
Thank you !