Hello,
I am trying to integrate Logstash with Kafka. Logstash works very well with Kafka as an output, but when configured as input, it doesnt work.
In logs it seems that communication between Kafka and Logstash works well and that I am listening on Kafka topic, but nothing comes through.
I can write to a topic, but cannot read from a topic with Logstash.
My input is as follows:
input {
kafka {
bootstrap_servers => "10.20.30.40:9092"
topics => "verified-topic"
}
}
Is there any reason this doesnt work? I tried Logstash version 6.0 and 5.5.0.
Kafka plugin version is:
bin/logstash-plugin list --verbose logstash-input-kafka
logstash-input-kafka (5.1.11)
In logs I see:
[2017-12-20T13:32:07,525][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 0.10.0.1
[2017-12-20T13:32:07,525][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : a7a17cdec9eaa6c5
[2017-12-20T13:32:07,659][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] Discovered coordinator name-of-server:9092 (id: 2147483647 rack: null) for group logstash.
[2017-12-20T13:32:07,661][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] Revoking previously assigned partitions [] for group logstash
[2017-12-20T13:32:07,661][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] (Re-)joining group logstash
[2017-12-20T13:32:07,670][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] Successfully joined group logstash with generation 15
[2017-12-20T13:32:07,670][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] Setting newly assigned partitions [verified-topic-0] for group logstash