How to set multiple hosts in elasticsearch-output via environment variable?

Elastic Stack Logstash
1

Hi,

I am trying to configure elasticsearch hosts via environment variable to be used in elasticsearch-output plugin. I want to set them as array for loadbalancing in the multi node cluster without the need for an external loadbalancer.

Here is my try:

output
{
        elasticsearch
        {
               
                hosts                         => [${ES_HOSTS}]
                ssl                           => "${USE_ES_SSL}"
                cacert                        => "${ES_CA_CERT_PATH}"
                ssl_certificate_verification  => "${USE_ES_OUTPUT_SSL_CERT_VERIFICATION}"

                # credentials are fetched from envrionment or logstash-keystore

                user            => "${LOGSTASH_USER}"
                password        => "${LOGSTASH_PASSWORD}"

                index           => "%{[@metadata][indexName]}"
        }
}

I've tried to set ES_HOSTS the following ways:

export ES_HOSTS='"https://server1:9200"'
echo $ES_HOSTS
"https://server1:9200"

export ES_HOSTS='\"https://server1:9200\"'
echo $ES_HOSTS
\"https://server1:9200\"

export ES_HOSTS='\""https://server1:9200"\"'
echo $ES_HOSTS
\""https://server1:9200"\"

export ES_HOSTS='""https://server1:9200""'
echo $ES_HOSTS
""https://server1:9200""

But I always get this error when I test the configuration with logtash's -t option:

[FATAL] 2020-01-13 11:39:25.118 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, ", ', -, [, {, ] at line 16, column 15 (byte 239) after output
{
        elasticsearch
        {
                hosts           => [
[ERROR] 2020-01-13 11:39:25.126 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

If that one is working for one server, then I want to be able to set something like this:
"https://server1:9200","https://server2:9200","https://server3:9200"

And if there is an external loadbalancer available, then I can just set it to the loadbalancer's "LB_IP:LB:PORT". So I would be flexible and more independent from the runtime environment without needing to change the pipeline between different environments.

Thanks, Andreas

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.