Multiple pipelines with Kafka input

Elastic Stack Logstash
1

When using the Kafka input plugin is it possible to assign different topics to their own pipeline in Logstash?

e.g with the Beats input we can setup different input configs to get Logstash to listen on different ports. Is a similar config possible with the Kafka input so that we can associate specific topics to specific Logstash ports/pipelines?

2

I think you would have to have different inputs, which can then trivially be in different pipelines.

3

Thanks Badger that's a good idea.

Do you know of any way to specify the individual pipeline ports on the kafka plugin config? I'm assuming logstash would need to consume each topic through a separate port in order to route each topic to its own pipeline

4

With the kafka input you give it a list of brokers and an array of topics and all the port assignments happen under the covers.

I don't have kafka set up right now, so I cannot test this, but from what I remember if you give one input topicA and another input topicB it should just work

5

OK we'll just try it out, see what happens. Not sure if 'under the cover' port assignments would work for us as our security guys like to lock down the firewall rules

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.