Not able to consume input from a kafka-topic

Elastic Stack Logstash
1

Hi, I am running logstash 2.3.4. I have kafka- 0.9 as the broker.

I also installed the logstash-kafka-input plugin using the matrix given on the github. 0.9 0.9 2.0.0 - 2.3.x 3.x.x
using the following command:
/opt/logstash/bin/logstash-plugin install --version 3.0.3 logstash-input-kafka

my input.conf looks like this:

input {
kafka {
bootstrap_servers => 'localhost:9092'
topics => 'logstash_2'
}
}

output {
elasticsearch {
hosts => ["localhost"]
}
}

on running /opt/logstash/bin/logstash -f kafka-conf.conf --verbose I see no error but unfortunately I can not see anytopic on my elasticsearch/kibana web-interface.

Any pointers will be greatly helpful.

2

Are you putting more data into the topic? The auto_offset_reset is mandatory. Try auto_offset_reset => "earliest".

Also check your offsets to confirm how the group is behaving. http://kafka.apache.org/090/documentation.html#basic_ops_consumer_lag

3

auto_offset_reset => "earliest" works as expected .

My detailed config:

  1. logstash-2.3.4 (downloaded with all plugins bundled)
  2. kibana-4.5.4
  3. elasticsearch-2.3.5
  4. kafka 0.9

As long as consumer-group is concerned I expected "logstash" to be the output of consumer groups, but it was throwing error on zookeeper-end. Including explicit group_id made it functional.

input {
     kafka {
             zk_connect => 'localhost:2181'
             topic_id => 'logstash_topic'
             group_id => "kafka_consumer"
             auto_offset_reset => "smallest"
     }
 }

 output {
     elasticsearch {
                 hosts => ["localhost"]
             }
 }

Another confusion, how can I check the installed version of a plugin in logstash-plugin list ?

bin/logstash-plugin list --verbose

4

That appears correct according to example 2 here: Working with plugins | Logstash Reference [8.11] | Elastic