Planned to upgrade my ELK setup to 5.x along with that I tried to add kafka message queue between forwarder and logstash server on the new environment.
There is no issues on forwarders sent message to kafka. I verified it through kafka consumer script.
But when I pull message from kafka using logstash-kafka-input plugin, its not writing any messages . Should I need to add/enable any specific settings in my configuration ?.
my logstash simple config :
input {
kafka{
topics => ["weblogs"]
bootstrap_servers => "10.11.12.202:9092"
}
}
output {
stdout { codec => rubydebug }
file {
path => "/tmp/stdout.log"
}
}
Kafka Consumer script output on the topic :
[root@logstash kafka_2.11-0.10.2.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic weblogs
10.11.12.169 - - [27/Apr/2017:13:50:26 +0000] "GET / HTTP/1.0" 302 287 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"
192.168.10.11 - - [27/Apr/2017:13:50:29 +0000] "GET / HTTP/1.0" 302 281 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"
10.11.12.169 - - [27/Apr/2017:13:50:29 +0000] "GET / HTTP/1.0" 302 291 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"
10.11.12.169 - - [27/Apr/2017:13:50:32 +0000] "GET / HTTP/1.0" 302 289 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)"