Capture environment variable

Elastic Stack Beats
1

Is it possible (or in the plans) to dynamically capture environment variable within a prospects entry or in tags on input section? Is there another way to do this instead?

We have a lot of instances and its not feasible to bake the config with hardcoded values but it would be great to reference an environment variable to get those details.

Thanks,

Sijis

2

@sijis We discussed in the past if we should allow environment varialbes in the config. This would mean they are interpreted on startup but then not changed anymore. Would this sufficient for you?

We didn't reach a conclusion here yet but if that is what you need you can already do it know as there are several scripts out there that can replace your environment variables inside a yaml file.

3

@ruflin Reading the environment variable on startup would be sufficient for our use case.

Are these scripts within filebeat or from other sources? If they are external, is there one you suggest to use?
I would hate to go through the script route (sort of sounds like a hack) and then the functionality be added to filebeat (via a plugin or core).

4

@sijis There is not one recommended way as it also depends on your environment. You could use sed to replace the variables and generate the actual config file always before startup. Or you could use jinia in combination with ansible.

Golang offers a function to expand environment variables, but we didn't add it on purpose as we think this could also have unexpected side affects and would not necessarly work on all environments: https://golang.org/pkg/os/#ExpandEnv

5

Hi Ruflin

It is possible to add runtime variable for the filebeat prospectors

like

path
-/opt/logs/*/operational.log.$VAR we want the date should be appended dynamically by the environment variables.

6

expanding environment variables is available in current 1.2 release. For variable expansion use ${VAR}.

7

It is really worked well.
-opt/logs/*/operationa.log.${MY_DATE} after restart it is picking up thelatest file with out any problem.