How to configure Logstash to output to dynamic list of kafka bootstrap servers?

[furiabhavesh (Bhavesh Furia)]

We need to configure logstash to output to multiple kafka brokers whose list needs to be dynamic.

We need to pass this list of kafka hosts as follows:

docker run -e BOOTSTRAP_SERVERS="host1:port1,host2:port2,hostn:portn"

and my output block is configured as below:

output {
  kafka {
    bootstrap_servers => "${BOOTSTRAP_SERVERS}"
    topic_id  => "${KAFKA_TOPIC_ID}"
  }
}

I noticed that Logstash config does not accept env variable if it contains colon ( : ) and fallbacks to default value of env variable.

Is it expected that we create N env variables for adding N no. of hosts or is there any other work around for this ?

[xeraa]

Others might run into the same issue, so quickly writing down the answer for the problem would be great

[furiabhavesh (Bhavesh Furia)]

It was not the case that logstash does not accept : in environment variable's values. It was just that I had a default value defined in keystore file which was always taking preference.

I am able to use only 1 env variable to pass multiple kafka bootstrap servers as string.

[ethrbunny]

Another approach would be a round-robin DNS entry that encompasses all of your kafka brokers.

We're using powerdns which supports LUA scripts. This gives us a nice spread between brokers.