What is the recommended way to dynamically tell logstash to which elasticsearch hosts logstash should connect to?
I have several hosts where I run logstash. However for some hosts I need the data to get to cluster A and for some hosts I need the data to be sent to cluster B.
What I used to do prior to 6.x is dynamically generate the output hosts (from a self-configured central config where I can pick corresponding cluster nodes excluding master dedicated ones) and append it to logstash.conf using the --config.string option. However, in 6.x --config.string is no longer appended so I was wondering maybe I have been doing it the non-optimum way all this time.
In 6.x, I can probably combine logstash.conf and --config.string into either a long string or write the same string into a file then passing it to bin/logstash. However, I was wondering if there is another better way?
Maintain different sets of configuration files for different hosts.
Use a configuration management tool (Ansible, Chef, Puppet, ...) to generate the configuration files with a templating language that for example supports variable substitution.
Yeah, luckily for me, the hosts already have the metadata that I need readily available on them. My wrapper start up script will expose the metadata via env vars that will be referenced inside logstash.conf.
Also tried export ES_HOSTS='[ "host1", "host2" ]' pass that to hosts => "${ES_HOSTS}" but this messes up the syntax by the looks of it.
If this hasn't been implemented, as a workaround, would it be a good idea to use ruby to generate a list from the env var and then save it into a meta field that would be referenced in the output plugin (granted that that ruby code would run on each event)?
Since the issue is open I have no reason to believe that it has been implemented.
If this hasn't been implemented, as a workaround, would it be a good idea to use ruby to generate a list from the env var and then save it into a meta field that would be referenced in the output plugin (granted that that ruby code would run on each event)?
I'm pretty sure the hosts option doesn't parse %{field} references, in which case your proposal won't work.
Any rough timeline for when that ticket is going to be done - just wondering cos it's been open for a year and I just wanna gauge whether I should wait or start on a different approach now.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.