Using list environment variables in filebeat config

[ApsOps (Amanpreet Singh)]

Previous discussion at Interpolate list from environment variable in filebeats and https://github.com/elastic/beats/pull/3045

I built from master, so this^ PR is included.

I'm using this config:

output.logstash:
    hosts: '${LOGSTASH_HOSTS:"logstash:5043"}'
    timeout: 15

Using LOGSTASH_HOSTS="host1:5043,host2:5043" gives an error:

failed to initialize logstash plugin as output: can not convert 'object' into 'string' accessing 'output.logstash.hosts'

---

In v1.2.x, this works with LOGSTASH_HOSTS="'host1:4083','host2:4084'" and following config:

output.logstash:
    hosts: [${LOGSTASH_HOSTS:'logstash:5043'}]

What is the correct way to using these type of environment variables now?

[steffens (Steffen Siering)]

Hi,

Thanks for continuing the discussion here

Doing some more tests it seems your hitting a bug in go-ucfg itself. A bug fix should be available in this PR.

You can try (and confirm fix works) by checking out the branch and recompile. Steps to test the PR with beats:

# 1. remove the vendored  go-ucfg
$ cd $GOPATH/github.com/elastic/beats/vendor/github.com/elastic
$ rm -fR go-ucfg

# 2. clone and check out go-ucfg branch including fix
$ cd $GOPATH/github.com/elastic
$ git clone https://github.com/urso/go-ucfg.git
$ cd go-ucfg
$ git checkout enh/parse

# 3. rebuild filebeat
$ cd $GOPATH/github.com/elastic/beats/filebeat
$ make

The PR includes new tests, checking for your exact use-case: https://github.com/urso/go-ucfg/blob/enh/parse/structs_test.go#L54

With this change config should be:

filebeat.yml

output.logstash:
  hosts: '${LOGSTASH_HOSTS:localhost:5043}'

Using LOGSTASH_HOSTS="host1:5043,host2:5043" should make it work.

In 5.x support for env-vars has been changed and expanded. In 1.x release, env variables have been expanded like string-templates, potentially resulting in YAML parsing errors (with potentially wrong line number). Plus, silently dropping configs if environment variable was missing. In 5.x variable expansion was introduced and more sophisticated variable parsing from configuration itself (after parsing). Variable expansion + parsing + support for overwriting settings in other config files or from command line + referencing other settings in any config file or environment variable + custom error message unfortunately doesn't mix well with plain template like text replacement.

You can for example print an error message if not logstash output is set:

output.logstash:
  hosts: '${LOGSTASH_HOSTS:?No logstash host configured}'

and overwrite the hosts from command line -E output.logstash.hosts=host1:5043,host2:5043

Or have a configure default host:

default.host=${TEST_HOST:localhost}

output.logstash:
  hosts: '${LOGSTASH_HOSTS:${default.host}:5043}'

and change default host from command line using -E default.host=other-host.

The 5.x config file format is documented here: https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format.html

Unfortunately the section on environment variable support is still lacking

[ApsOps (Amanpreet Singh)]

Thanks a lot for helping out.

Building go-ucfg fix from the branch works!

@steffens

You can for example print an error message if not logstash output is set:

output.logstash:
hosts: '${LOGSTASH_HOSTS:?No logstash host configured}'

and overwrite the hosts from command line -E output.logstash.hosts=host1:5043,host2:5043

I'm not able to get this^ working though. It fails, but the custom error message is not printed.

[steffens (Steffen Siering)]

Thanks for testing.

You're exactly hitting this line:

    // TODO: improve error message

I just pushed another commit to the branch addressing the TODO.

It's supposed to print:

ERR failed to initialize logstash plugin as output: No logstash host configured accessing 'output.logstash.hosts' (source:'filebeat.yml')

Doing this (testing with metricbeat) works for me (metricbeat starts up and uses both hosts):

./metricbeat -e -v -c metricbeat.yml -E output.logstash.hosts=localhost:1234,localhost:5678

Maybe you have to put the hosts in quotes (I'm using the zsh - z shell)? Also do not forget the space between -E and the setting to overwrite hosts from CLI.

For testing I often have another config file with outputs. E.g. I run filebeat like this:

filebeat -e -v -c filebeat.yml -c outputs.yml

[ApsOps (Amanpreet Singh)]

Awesome! The error message is shown correctly with the update.

Thanks again