Advice on dynamically configuring logstash 6.2 output plugin

Hi Team,

What is the recommended way to dynamically tell logstash to which elasticsearch hosts logstash should connect to?

I have several hosts where I run logstash. However for some hosts I need the data to get to cluster A and for some hosts I need the data to be sent to cluster B.

What I used to do prior to 6.x is dynamically generate the output hosts (from a self-configured central config where I can pick corresponding cluster nodes excluding master dedicated ones) and append it to logstash.conf using the --config.string option. However, in 6.x --config.string is no longer appended so I was wondering maybe I have been doing it the non-optimum way all this time.

In 6.x, I can probably combine logstash.conf and --config.string into either a long string or write the same string into a file then passing it to bin/logstash. However, I was wondering if there is another better way?

Cheers,

I can think of a couple of options:

  • Maintain different sets of configuration files for different hosts.
  • Use a configuration management tool (Ansible, Chef, Puppet, ...) to generate the configuration files with a templating language that for example supports variable substitution.
  • Use Logstash's support for referencing environment variables (https://www.elastic.co/guide/en/logstash/current/environment-variables.html).

Thanks Magnus for your suggestions.

I think I like the environment variables approach the best as it would allow me to have one central config that is dynamic.

Maintaining different sets of configs could be a bit cumbersome just cos there would be quite a few sets to manually maintain.

I am using ansible already but I am trying to avoid "extra couplings" involved with config generations.

Cheers!

I am using ansible already but I am trying to avoid "extra couplings" involved with config generations.

And how are you going to differentiate the environment variables between hosts?

Hey Magnus,

Yeah, luckily for me, the hosts already have the metadata that I need readily available on them. My wrapper start up script will expose the metadata via env vars that will be referenced inside logstash.conf.

Hi Team,

There's an open ticket for making array env vars work: https://github.com/elastic/logstash/issues/6366, do you know if this has been implemented?

I tried several ways of exporting an env var as a list but logstash isn't treating the env var as an array but a string instead, eg:

export ES_HOSTS='host1,host2'

Then that would be referenced here:

output {
  elasticsearch {
    hosts => [ "${ES_HOSTS}" ]
  }
}

Also tried export ES_HOSTS='[ "host1", "host2" ]' pass that to hosts => "${ES_HOSTS}" but this messes up the syntax by the looks of it.

If this hasn't been implemented, as a workaround, would it be a good idea to use ruby to generate a list from the env var and then save it into a meta field that would be referenced in the output plugin (granted that that ruby code would run on each event)?

There's an open ticket for making array env vars work: Logstash 2.4.1/5.0.x: Environments Variables parsed as Lists/Arrays for Logstash Configuration · Issue #6366 · elastic/logstash · GitHub, do you know if this has been implemented?

Since the issue is open I have no reason to believe that it has been implemented.

If this hasn't been implemented, as a workaround, would it be a good idea to use ruby to generate a list from the env var and then save it into a meta field that would be referenced in the output plugin (granted that that ruby code would run on each event)?

I'm pretty sure the hosts option doesn't parse %{field} references, in which case your proposal won't work.

That's unfortunate.

Any rough timeline for when that ticket is going to be done - just wondering cos it's been open for a year and I just wanna gauge whether I should wait or start on a different approach now.

I really hope the feature gets done soon as filebeat has it: https://www.elastic.co/guide/en/beats/filebeat/current/using-environ-vars.html and thought logstash would also have it.

Any rough timeline for when that ticket is going to be done

I have no idea.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.