How to set multiple hosts in elasticsearch-output via environment variable?

Hi,

I am trying to configure elasticsearch hosts via environment variable to be used in elasticsearch-output plugin. I want to set them as array for loadbalancing in the multi node cluster without the need for an external loadbalancer.

Here is my try:

output
{
        elasticsearch
        {
               
                hosts                         => [${ES_HOSTS}]
                ssl                           => "${USE_ES_SSL}"
                cacert                        => "${ES_CA_CERT_PATH}"
                ssl_certificate_verification  => "${USE_ES_OUTPUT_SSL_CERT_VERIFICATION}"

                # credentials are fetched from envrionment or logstash-keystore

                user            => "${LOGSTASH_USER}"
                password        => "${LOGSTASH_PASSWORD}"

                index           => "%{[@metadata][indexName]}"
        }
}

I've tried to set ES_HOSTS the following ways:

export ES_HOSTS='"https://server1:9200"'
echo $ES_HOSTS
"https://server1:9200"

export ES_HOSTS='\"https://server1:9200\"'
echo $ES_HOSTS
\"https://server1:9200\"

export ES_HOSTS='\""https://server1:9200"\"'
echo $ES_HOSTS
\""https://server1:9200"\"

export ES_HOSTS='""https://server1:9200""'
echo $ES_HOSTS
""https://server1:9200""

But I always get this error when I test the configuration with logtash's -t option:

[FATAL] 2020-01-13 11:39:25.118 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, ", ', -, [, {, ] at line 16, column 15 (byte 239) after output
{
        elasticsearch
        {
                hosts           => [
[ERROR] 2020-01-13 11:39:25.126 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

If that one is working for one server, then I want to be able to set something like this:
"https://server1:9200","https://server2:9200","https://server3:9200"

And if there is an external loadbalancer available, then I can just set it to the loadbalancer's "LB_IP:LB:PORT". So I would be flexible and more independent from the runtime environment without needing to change the pipeline between different environments.

Thanks, Andreas

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.