Logstash: dynamic topic based on value of field

Elastic Stack Logstash
1

I'm trying to configure Logstash with Kafka output. Basically I want to put data in different Kafka topics based on a field.
So in example below, I have a field name in the data. All documents should be posted to the topic per name.
E.g. if name is test1, topic should be topic_test1, for test2 it should be topic_test2 etc...

output {
  kafka {
    id => "topic-kafka"
    bootstrap_servers => "kafka.test.be:9092"
    topic_id => "topic_%{name}"
    codec => json
  }
}

I tried to use the sprintf format like above, but that doesn't work.

[2022-07-29T11:51:52,502][ERROR][org.apache.kafka.clients.Metadata][topic-kafka] [Producer clientId=producer-1] Metadata response reported invalid topics [topic_%{name}]

I'm using logstash version v7.16.2

Any ideas how we can configure this?

Thanks!

2

This is supported and should work if you have the name field in your document in logstash.

Can you share what is the output in logstash when you got that error? You can add an extra stdout output to see the document that logstash is sending.

3

Indeed, there was a mistake in my logstash. This caused that the name field was not existing, therefore the error was popping up.
Thanks to point that out.
So the output config as I posted was correct after all.

Thanks, ticket can be closed.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.