Advice on dynamically configuring logstash 6.2 output plugin

[ld_pvl]

Hi Team,

What is the recommended way to dynamically tell logstash to which elasticsearch hosts logstash should connect to?

I have several hosts where I run logstash. However for some hosts I need the data to get to cluster A and for some hosts I need the data to be sent to cluster B.

What I used to do prior to 6.x is dynamically generate the output hosts (from a self-configured central config where I can pick corresponding cluster nodes excluding master dedicated ones) and append it to logstash.conf using the --config.string option. However, in 6.x --config.string is no longer appended so I was wondering maybe I have been doing it the non-optimum way all this time.

In 6.x, I can probably combine logstash.conf and --config.string into either a long string or write the same string into a file then passing it to bin/logstash. However, I was wondering if there is another better way?

Cheers,

[magnusbaeck (Magnus Bäck)]

I can think of a couple of options:

• Maintain different sets of configuration files for different hosts.
• Use a configuration management tool (Ansible, Chef, Puppet, ...) to generate the configuration files with a templating language that for example supports variable substitution.
• Use Logstash's support for referencing environment variables (https://www.elastic.co/guide/en/logstash/current/environment-variables.html).

[ld_pvl]

Thanks Magnus for your suggestions.

I think I like the environment variables approach the best as it would allow me to have one central config that is dynamic.

Maintaining different sets of configs could be a bit cumbersome just cos there would be quite a few sets to manually maintain.

I am using ansible already but I am trying to avoid "extra couplings" involved with config generations.

Cheers!

[magnusbaeck (Magnus Bäck)]

I am using ansible already but I am trying to avoid "extra couplings" involved with config generations.

And how are you going to differentiate the environment variables between hosts?

[ld_pvl]

Hey Magnus,

Yeah, luckily for me, the hosts already have the metadata that I need readily available on them. My wrapper start up script will expose the metadata via env vars that will be referenced inside logstash.conf.

[ld_pvl]

Hi Team,

There's an open ticket for making array env vars work: https://github.com/elastic/logstash/issues/6366, do you know if this has been implemented?

I tried several ways of exporting an env var as a list but logstash isn't treating the env var as an array but a string instead, eg:

export ES_HOSTS='host1,host2'

Then that would be referenced here:

output {
  elasticsearch {
    hosts => [ "${ES_HOSTS}" ]
  }
}

Also tried export ES_HOSTS='[ "host1", "host2" ]' pass that to hosts => "${ES_HOSTS}" but this messes up the syntax by the looks of it.

If this hasn't been implemented, as a workaround, would it be a good idea to use ruby to generate a list from the env var and then save it into a meta field that would be referenced in the output plugin (granted that that ruby code would run on each event)?

[magnusbaeck (Magnus Bäck)]

There's an open ticket for making array env vars work: Logstash 2.4.1/5.0.x: Environments Variables parsed as Lists/Arrays for Logstash Configuration · Issue #6366 · elastic/logstash · GitHub, do you know if this has been implemented?

Since the issue is open I have no reason to believe that it has been implemented.

If this hasn't been implemented, as a workaround, would it be a good idea to use ruby to generate a list from the env var and then save it into a meta field that would be referenced in the output plugin (granted that that ruby code would run on each event)?

I'm pretty sure the hosts option doesn't parse %{field} references, in which case your proposal won't work.

[ld_pvl]

That's unfortunate.

Any rough timeline for when that ticket is going to be done - just wondering cos it's been open for a year and I just wanna gauge whether I should wait or start on a different approach now.

I really hope the feature gets done soon as filebeat has it: https://www.elastic.co/guide/en/beats/filebeat/current/using-environ-vars.html and thought logstash would also have it.

[magnusbaeck (Magnus Bäck)]

Any rough timeline for when that ticket is going to be done

I have no idea.