# Kafka output plugin not working

**URL:** <https://discuss.elastic.co/t/kafka-output-plugin-not-working/147187>\
**Category:** Logstash\
**Created:** [September 4, 2018, 10:18am UTC](https://discuss.elastic.co/t/kafka-output-plugin-not-working/147187 "2018-09-04T10:18:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![asatsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asatsi/32/28417_2.png) [@asatsi](https://discuss.elastic.co/u/asatsi)\
**Post date:** [September 4, 2018, 10:18am UTC](https://discuss.elastic.co/t/kafka-output-plugin-not-working/147187/1 "2018-09-04T10:18:37Z")

</div>

Hi, I am trying to push logs onto a kafka topic. Has anybody successfully used Kafka output plugin in logstash?

Tried with logstash version: 6.3.2/6.4.0 but same result.

Below is my config file:  
input {  
file  
{  
path =\> "/tmp/sat.txt"  
}  
}

filter{}

output  
{  
stdout { codec =\> rubydebug }  
kafka {  
codec=\> "plain"  
topic\_id =\> "prod-esb2"  
bootstrap\_servers =\> "[kafka.mydomain.com:9092](http://kafka.mydomain.com:9092)"  
}  
}

This is the error I see in logstash in debug mode:  
[2018-09-04T16:29:32,961][DEBUG][org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Initialize connection to node [kafka.mydomain.com:9092](http://kafka.mydomain.com:9092) (id: -1 rack: null) for sending metadata request  
[2018-09-04T16:29:32,961][DEBUG][org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Initiating connection to node [kafka.mydomain.com:9092](http://kafka.mydomain.com:9092) (id: -1 rack: null)  
[2018-09-04T16:29:32,999][DEBUG][org.apache.kafka.common.metrics.Metrics] Added sensor with name node--1.bytes-sent  
[2018-09-04T16:29:33,004][DEBUG][org.apache.kafka.common.metrics.Metrics] Added sensor with name node--1.bytes-received  
[2018-09-04T16:29:33,006][DEBUG][org.apache.kafka.common.metrics.Metrics] Added sensor with name node--1.latency  
[2018-09-04T16:29:33,007][DEBUG][org.apache.kafka.common.network.Selector] [Producer clientId=producer-1] Created socket with SO\_RCVBUF = 32768, SO\_SNDBUF = 131072, SO\_TIMEOUT = 0 to node -1  
[2018-09-04T16:29:33,176][DEBUG][org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Completed connection to node -1. Fetching API versions.  
[2018-09-04T16:29:33,177][DEBUG][org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Initiating API versions fetch from node -1.  
[2018-09-04T16:29:33,190][DEBUG][org.apache.kafka.common.network.Selector] [Producer clientId=producer-1] Connection with [kafka.mydomain.com/10.250.191.80](http://kafka.mydomain.com/10.250.191.80) disconnected  
java.io.EOFException: null  
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124) ~[kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93) ~[kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235) ~[kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196) ~[kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:557) ~[kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:495) [kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.common.network.Selector.poll(Selector.java:424) [kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) [kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) [kafka-clients-1.1.0.jar:?]  
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) [kafka-clients-1.1.0.jar:?]  
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_45]  
[2018-09-04T16:29:33,198][DEBUG][org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Node -1 disconnected.  
[2018-09-04T16:29:33,200][DEBUG][org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Give up sending metadata request since no node is available

---

<div class="post-metadata">

**Author:** ![asatsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asatsi/32/28417_2.png) [@asatsi](https://discuss.elastic.co/u/asatsi)\
**Post date:** [September 5, 2018, 5:38am UTC](https://discuss.elastic.co/t/kafka-output-plugin-not-working/147187/2 "2018-09-05T05:38:40Z")

</div>

OK, I seem to have made some progress here. I copied the kafka-clients.jar (0.9.0) from the broker to the logstash vendor lib directory. And also modified the ruby file logstash-outpu  
t-kafka\_jars.rb to pick up the 0.9.0 version of the jar. This looks to be working.

But clearly the backward compatibility is broken OOTB!

@logstash @elastic team, could you please confirm whether this is a bug, so that I can open bug on the git repo?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 3, 2018, 5:38am UTC](https://discuss.elastic.co/t/kafka-output-plugin-not-working/147187/3 "2018-10-03T05:38:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.