Logstash-kafka for multiple topics

Elastic Stack Logstash
1

Hi! There's been some interest in the past for logstash to support multple kafka topics as inputs. At the time, the only solution was to have one topic per input. Has there been additional progress on that? Is it possible to specify multiple topics as inputs? Thanks!

2

There is support for multiple queues per input. check out the white_list and black_list settings. They take java compatible regular expressions and will consume all queues that match the pattern. So you could do "topicA|topicB|logstash-*" and it would consume a queue named topicA, topicB and any that are prefixed with the name logstash-, ie logstash-tomcatlogs. etc.

1 Like
3

Looks like what I need! I will check it out. Thanks, @Joe_Lawson!

4

The addition of white_list and black_list is great -- thanks! One question -- is there a way to know which topic each message comes from when using white_list/black_list?

I have so far done this by generating a logstash config from a template and create multiple input clauses -- in each generated input clause, I have a unique topic_id and corresponding add_field statement like this:

    zk_connect => "localhost:3011,localhost:3012,localhost:3013"
    topic_id => "log_some.host.domain_heartbeat"
    reset_beginning => "false"
    group_id => "logstash-heartbeat"

    type => "logmonitor"
    add_field => {"kafka_topic_id" => "log_some.host.domain_heartbeat"}

This way, in the filter section, we could access kafka_topic_id to know the topic from whence the message came.

I'd love to use white_list/black_list instead of the many input clauses contraption I have now -- Is there some way to know which topic each message comes from when using white_list/black_list?

5

Just use decorate_events => "true" from
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-decorate_events

See:

1 Like
6

You're very kind not to have said RTFM -- sorry, I glossed right over that in the doc.

I'll give this a try shortly.

Thanks Joe!