Managing beats and template versions

[jreichman]

Hi,

I'm trying to work out the best way to manage differing versions of beats for an elastic search instance.

If I output from filebeat to logstash, then to elastic search, I miss out on automatic index template creation from filebeat, per agent version, and need to do this manually (e.g. manually load an index template for filebeat-7.9.2-*). This is a pain with several differing versions across an environment with different people managing them.

I can connect the filebeat instance directly to elastic search, bypassing logstash, which allows me to automatically load the index template, though now the filebeat instance has permissions to update index templates in the elastic search instance, which seems like a permission that I shouldn't allow for systems potentially managed by different people. The documentation also indicates that I should have a separate role for setup: https://www.elastic.co/guide/en/beats/filebeat/current/privileges-to-setup-beats.html

How do you properly manage index template versions when you have unknown versions of beats in the environment and don't want to hand out over privileged credentials to each beat instance?

Hope this makes sense.

Jesse

[mtojek (Marcin Tojek)]

@jreichman

though now the filebeat instance has permissions to update index templates in the Elasticsearch instance, which seems like a permission that I shouldn't allow for systems potentially managed by different people

I don't think there is an easy way to support such procedure (with logstash in the middle). Did you try to use standard filebeat setup and keep only index templates, I mean strip all filebeat permissions later on?

[andrewkroh]

@jreichman

How do you properly manage index template versions when you have unknown versions of beats in the environment and don't want to hand out over privileged credentials to each beat instance?

Here's one option if you can run the setup command once when you introduce a new version.

You can control the versions that you allow by restricting the writer role's index permissions to specific versions. When you are ready to allow a new version you first use the setup role to create the index template and establish the index lifecycle management policy. After the setup is complete you can then modify the writer role to allow writes to the filebeat-7.X.Y index.