How to configure Logstash to output to dynamic list of kafka bootstrap servers?

We need to configure logstash to output to multiple kafka brokers whose list needs to be dynamic.

We need to pass this list of kafka hosts as follows:

docker run -e BOOTSTRAP_SERVERS="host1:port1,host2:port2,hostn:portn"

and my output block is configured as below:

output {
  kafka {
    bootstrap_servers => "${BOOTSTRAP_SERVERS}"
    topic_id  => "${KAFKA_TOPIC_ID}"
  }
}

I noticed that Logstash config does not accept env variable if it contains colon ( : ) and fallbacks to default value of env variable.

Is it expected that we create N env variables for adding N no. of hosts or is there any other work around for this ?

Others might run into the same issue, so quickly writing down the answer for the problem would be great :slight_smile:

It was not the case that logstash does not accept : in environment variable's values. It was just that I had a default value defined in keystore file which was always taking preference.

I am able to use only 1 env variable to pass multiple kafka bootstrap servers as string.

Another approach would be a round-robin DNS entry that encompasses all of your kafka brokers.

We're using powerdns which supports LUA scripts. This gives us a nice spread between brokers.

1 Like