Hi,
I'm looking at using kafka as a message queue in our elk environment. I have two logstash instances running, the first takes inbound json and outputs to kafka while the second reads from kafka and outputs to elasticsearch.
Right now I have one message sitting in kafka waiting to be consumed:
$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic logstash-logs --time -1
logstash-logs:0:1
So, the kafka output looks to be working while the kafka input isn't. Here is my input config:
input {
kafka {
zk_connect => "testbroker:2181"
topic_id => "logstash-logs"
tags => "kafka-input"
queue_size => 20
fetch_message_max_bytes => 1048576
consumer_threads => 1
consumer_id => "ls-indexer"
}
} # end input
I've tested connectivity to the zookeeper port and that looks fine. And I've tested the elasticsearch output successfully via another logstash instance which passes messages straight through. So, as I'm relatively new to logstash, could someone help me with troubleshooting the failure to consume/send to elasticsearch please?
Regards,
David