Logstash Kafka Input plugin : no message read from kafka topic

Elastic Stack Logstash
1

Planned to upgrade my ELK setup to 5.x along with that I tried to add kafka message queue between forwarder and logstash server on the new environment.

There is no issues on forwarders sent message to kafka. I verified it through kafka consumer script.

But when I pull message from kafka using logstash-kafka-input plugin, its not writing any messages . Should I need to add/enable any specific settings in my configuration ?.

my logstash simple config :

input {
kafka{
	topics => ["weblogs"]
	bootstrap_servers => "10.11.12.202:9092"
}
}

output {
  stdout { codec => rubydebug }
 file {
                 path => "/tmp/stdout.log"
           }
}

Kafka Consumer script output on the topic :

[root@logstash kafka_2.11-0.10.2.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic weblogs
10.11.12.169 - - [27/Apr/2017:13:50:26 +0000] "GET / HTTP/1.0" 302 287 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"
192.168.10.11 - - [27/Apr/2017:13:50:29 +0000] "GET / HTTP/1.0" 302 281 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"
10.11.12.169 - - [27/Apr/2017:13:50:29 +0000] "GET / HTTP/1.0" 302 291 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"
10.11.12.169 - - [27/Apr/2017:13:50:32 +0000] "GET / HTTP/1.0" 302 289 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"

2

Any idea ?. Due to this problem, I'm unable to upgrade my production env to latest version. can someone help me on this please ?

3

Can I create this issue in github ?. but I would like to know whether its a configuration related issue or something else. Thank you.

4

I have run the logstash on debug mode and paste the log here. Please someone help me to fix this issue.

https://paste.ubuntu.com/24630775/

5

Hi I am faced with the same problem , logstash version 5.3 ,
get nothing from kafka topic,did you solved it

thanks.

6

Yes i fixed the issue.

Thanks,
Gugan.K

7

The problem is not because of logstash input kafka plugin,
It's kafka issue. By default it doesnt allow remote connection. You should
enable it in kafka to allow remote connection

Thanks,
Gugan.K

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.