We have a set-up with sort of a hierarchical flow in the pipelines. Depending on application/format they're being routed to a different pipeline. All pipelines use the same Elasticsearch cluster for output, but in some cases a different ilm_rollover_alias so we can define different retention policies based on data sources.
Is there any sort of elegant solution to store the hosts & credential in one location, but specify the rollover alias in each individual pipe? I wouldn't want to go through dozens of pipelines just to change a token or hostnames...
I'd love some way to define environment variables in logstash.yml that could be used in all the pipelines. Environment variables would require you to re-create containers, the keystore is even more bothersome to work with...
Following your suggestion, we did some automation around Bitbucket and Jenkins to roll out configuration changes in a simple manner. Would be awesome if all we needed to do in order to add or remove elastic nodes from pipelines was a simple change to a single yml and maybe a container restart.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.