Logstash - any best practices for reusing elasitcsearch outputs across pipelines?

[mybyte (Michael)]

We have a set-up with sort of a hierarchical flow in the pipelines. Depending on application/format they're being routed to a different pipeline. All pipelines use the same Elasticsearch cluster for output, but in some cases a different ilm_rollover_alias so we can define different retention policies based on data sources.

Is there any sort of elegant solution to store the hosts & credential in one location, but specify the rollover alias in each individual pipe? I wouldn't want to go through dozens of pipelines just to change a token or hostnames...

[leandrojmp (Leandro Pereira)]

Maybe you can use the Logstash Keystore or Environment Variables.

In my case I use environment variables in the file /etc/sysconfig/logstash.

For example, in the file I will have the variables:

ES_USERNAME=username
ES_PASSWORD=password
ES_NODES="https://node01:9200 https://node02:9200 https://nodeN:9200"

Then in the output I would have:

output {
    elasticsearch {
        hosts => ["${ES_NODES}"]
        index => "index-name"
        http_compression => true
        user => '${ES_USERNAME}'
        password => '${ES_PASSWORD}'
    }
}

You could do the same with the logstash-keystore, I prefer the file with environment variables because it is easier to automate.

Also, any change on the values of the variables or keystore would require a logstash restart.

But, this way you would still need to edit the pipelines to change the ilm_rollover_alias.

Unfortunatelly the ilm_rollover_alias does not support dynamic variable substitution that would allow you to use a similar approach.

[mybyte (Michael)]

I'd love some way to define environment variables in logstash.yml that could be used in all the pipelines. Environment variables would require you to re-create containers, the keystore is even more bothersome to work with...

Following your suggestion, we did some automation around Bitbucket and Jenkins to roll out configuration changes in a simple manner. Would be awesome if all we needed to do in order to add or remove elastic nodes from pipelines was a simple change to a single yml and maybe a container restart.