Externalization of hosts settings in output

[sunilmchaudhari (Sunil Chaudhari)]

Hi,
I have a logstash indexing to 3 elasticsearch nodes.
The output filter is as below:
elasticsearch
{
hosts => [ "https://myHost9a.vsi.uat.dbs.com:9202", "https://myHost10a.vsi.uat.dbs.com:9202", "https://myHost11a.vsi.uat.dbs.com:9202" ]
index => "index_name_missing-%{+YYYY.MM.dd}"
timeout => 300
ssl => true

}

you can ignore the syntax and other settings. I am only concerned about the hosts setting.
In future, It may happen that I remove myHost11a.
So, If I have 3 such instances of logstash with 5 pipelines, then I have to change the output config from 3*5=15 output files.

is there any way, where I can mention some proxy or dummy names in actual config files and original host names will be there in some external file with key value pair.
Then I will change only the external file. I will remove the thir node from external file and restart logstash, OR logstash will auto reload it.
So that after restart logstash will not index to the "myHost11a".

So basically I want to know if there is possibility of externalization of the hosts setting, where instead of mentioning actual ES hosts I will give some dummy values which will be read from external txt file?

Regards,
Sunil.

[ylasri (Yassine LASRI)]

You can use environment-variables

[Badger]

@ylasri

You can use environment-variables

Are you sure about that? My recollection is that elasticsearch will treat a string that look like an array as a string, not an array, so it will not parse the hostname correctly.

[ylasri (Yassine LASRI)]

Hi @Badger, Thanks for comenting, i would agree with you for parameter that accept arrays
There is a specific PR for URI array that has been fixed and i guess that would adress this issue.

So Something like this should work

export ES_HOSTS="es1.example.com es2.example.com:9201 es3.example.com:9201"

output {
  elasticsearch {
    hosts => "${ES_HOSTS}"
  }
}

[Badger]

@ylasri

There is a specific PR for URI array that has been fixed

Excellent!

[sunilmchaudhari (Sunil Chaudhari)]

Hi @ylasri,
I already tried Env variable before I saw your reply. However, I tried this way with comma separated.

export ES_HOSTS="es1.example.com, es2.example.com:9201 es3.example.com:9201"

This was giving error. Current version I am using is, 7.5.0
May I will try without comma separated.
If that doesnt work, Let me know whether I have to upgrade.

regards,
Sunil.