How to configure Logstash to output to dynamic list of kafka bootstrap servers?

Elastic Stack Logstash
1

We need to configure logstash to output to multiple kafka brokers whose list needs to be dynamic.

We need to pass this list of kafka hosts as follows:

docker run -e BOOTSTRAP_SERVERS="host1:port1,host2:port2,hostn:portn"

and my output block is configured as below:

output {
  kafka {
    bootstrap_servers => "${BOOTSTRAP_SERVERS}"
    topic_id  => "${KAFKA_TOPIC_ID}"
  }
}

I noticed that Logstash config does not accept env variable if it contains colon ( : ) and fallbacks to default value of env variable.

Is it expected that we create N env variables for adding N no. of hosts or is there any other work around for this ?

2

Others might run into the same issue, so quickly writing down the answer for the problem would be great :slight_smile:

3

It was not the case that logstash does not accept : in environment variable's values. It was just that I had a default value defined in keystore file which was always taking preference.

I am able to use only 1 env variable to pass multiple kafka bootstrap servers as string.

4

Another approach would be a round-robin DNS entry that encompasses all of your kafka brokers.

We're using powerdns which supports LUA scripts. This gives us a nice spread between brokers.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.