Logstash and Kafka input issue

Elastic Stack Logstash
1

Hello,

I am trying to setup the Kafka input in Logstash but something is not working right. Any help would be appreciated. Thanks in advance.

Error Message on Server 2 in logstash.log

{:timestamp=>"2016-03-18T10:29:52.877000-0500", :message=>"kafka client threw exception, restarting", :exception=>#<KafkaError: Got ZkException: Unable to connect to zookeeper server within timeout: 6000>, :level=>:warn}

My setup is like this:

Server 1

  • logstash 2.2.2-1
  • Kafka 2.11-0.9.0.1

input {
udp {
port => 5544
type => "cisco-asa"
}
}
output {
kafka {
topic_id => "test"
}
}

Server 2

  • logstash 2.2.2-1

input {
kafka {
zk_connect => "Server1:2181"
topic_id => "test"
}
}

  • For zk_connect option on Server 2 I've tried using IP address and hostname of Server 1but both display same error above
  • Confirmed test topic is in Kafka
    /opt/kafka_2.11-0.9.0.1/bin/kafka-topics.sh --list --zookeeper localhost:2181
    test
  • Confirmed that Server 1 is listening on port 2181
  • Confirmed UDP input is being sent to Kafka test topic - /opt/kafka_2.11-0.9.0.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
  • Zookeeper properties has maxClientCnxns=0
2

This is a little embarrassing but it looks like I had the wrong subnet mask in my network config on Server 1. No longer seeing error in logstash logs.

1 Like
3

Glad it is working and thanks for following up!

1 Like
4

Hi,
Its not working for me.
I am able to inject data to kafka queue using logstash but unable to read.
My conf are same as HelpComputer mentioned.
Logstash starts but does not fetch data. No error is coming up.
Kafka, localstash and elasticsearch are running on my local system.
Any help is appreciated

5

Please start your own thread for this issue and provide more details about your configuration.

6

How to start new thread?

7

Use the "New Topic" button that is available in the top right corner on, e.g. the landing page.

8

created new topic:
"Logstash Input Plugin for kafka 0.9 is not working"