Logstash with multiple kafka inputs

Elastic Stack Logstash
1

I am trying to filter kafka events from multiple topics, but once all events from one topic has been filtered logstash is not able to fetch events from the other kafka topic. I am using topics with 3 partitions and 2 replications Here is my logstash confi

input {
kafka{
auto_offset_reset => "smallest"
consumer_id => "logstashConsumer1"
topic_id => "unprocessed_log1"
zk_connect=>"192.42.79.67:2181,192.41.85.48:2181,192.10.13.14:2181"
type => "kafka_type_1"
}
kafka{
auto_offset_reset => "smallest"
consumer_id => "logstashConsumer1"
topic_id => "unprocessed_log2"
zk_connect => "192.42.79.67:2181,192.41.85.48:2181,192.10.13.14:2181"
type => "kafka_type_2"
}
}
filter{
if [type] == "kafka_type_1"{
csv {
separator=>" "
source => "data"
}
}
if [type] == "kafka_type_2"{
csv {
separator => " "
source => "data"
}
}
}
output{
stdout{ codec=>rubydebug{metadata => true }}
}

2

Please edit your subject so that you are not answering the entire question there.

3

With two kafka inputs the messages are processed in parallel. Are you just at the end of the data?

4

With above configurations data of only one topic was processed. My expectation was to process all the data from both topics. Now I have used different consumer_id of both inputs and its working perfectly fine.

5

Curious. I would expect each to be able to use the same as long as the topic is different. Thanks for the update!