Beat name from file

[lendoly (Luis)]

Hello to all,

I already see that is possible to configure the beat name with Environment Variables, but I want to know if is possible to configure the beat name from a file.

Explanation:
I have 2 machines running on AWS, each one have two dockers (1 for filebeat and other for metricbeat) and the deploy is done by elastic benstalk (eb) and I can not pass dynamics env vars like the hostname of the host, so I can do two things:

• Set a static beat name
• Let the default hostname as beat name

If I set the beat name static, my both metricbeats will index metrics with the same beat name, so I can't monitor well the data and the status.

With the second option the hostname is the docker container hostname and is random generate by eb so the data is now correct for each machine, but I didn't know which data is from each machine in a quick way, I need to go to each machine and check the hostname of the docker container, not a big problem with 2 machines but with 4,5,6, etc. you can imagine ... and this is not really fast for critical situations.

So if I can set the beat name from a file I can share between container and host the /etc/hostname from the host and get a more verbose name for the beat.

Extra-info:
all is running on 5.0

PS: Sorry if the question is duplicated, but I didn't found nothing similar

[ruflin]

Two ideas:

• You can set the beat.name on startup with -E beat.name=beatname. Perhaps you can do some magic with this?
• Beats config can be chained since 5.0 by using multiple -c params. So if you use a volume, to mount a file, perhaps you could start the beat with the default config file point -c to it and a second -c for the config file with only beat.name inside?

Please be aware, didn't test both of the above, just some ideas that should work

[steffens (Steffen Siering)]

A third option is to configure an environment variable used in your configuration file.

[lendoly (Luis)]

Thanks man, both of them can be good solutions and I can work with that.

If I found a solution I will share it here, thanks again!

[lendoly (Luis)]

I tried, but is not so easy, because is running inside a docker and I don't found a way to set the environment variable on the startup of the docker.

One solution can be add it on the docker run but I can't touch the docker run command, because I'm doing the deploy by elastic beanstalk and it's done internally.

[lendoly (Luis)]

Hello to all, here the solution that I found, thanks to @ruflin for the idea.

Initial situation

We have a environment with 2 or more machines working on AWS and we are making the deploy with their service elastic beanstalk (eb) with docker.

Inside of the machines we are running some container, two of them for a beat service (in my case one for metricbeat* and other for filebeat) also a relevant info is that the beats services are running by supervisor.

*metericbeat recolect info from each container running and from the host.

Problem

When eb make the deploy it set a random hostanme/dockercontainerid/etc to the containers and in the configuration.yml for the beats. We didn't set a name for the beat, so when we see the info of the CPU on Kibana the info was correct, but was difficult to identify which info was from each machine and in each deploy the name change and we lost the historic of the info.

With that situation we think on set a default name for the beat, but this didn't work as we expected, all the info was mixed between beats, because if we make the deploy on 2 machines the metric beat of each machine have the same name, so in Kibana you see all mixed (if you filter by beat name obviously)

In that point we thought on set as name of the beat the type of beat and the hostname of the host machine of the dockers (filebeat-hostname and metricbeat-hostname) and make more easy know from where the info was coming.

First approach

Reading the documentation of beat, you can find that you can use environment variables on the configuration of the beats, so it sound easy to make something similar to:

beat.name: metricbeat-${HOST_HOSTNAME}

The idea was god, but problematic, if you thought on make that with this configuration I wish you good luck.

You can't modify the docker run for pass the env var that you want or set a hostname for the container because is done internally. AWS give some tools for pass to the machine statics env vars, but not dynamic one, so we was on the same dead-point as set it on the beat configuration because you cant set enviroment variables as $HOSTNAME.

Eb also allow to share folder between host and container and execute commands on the host before the odcker run is executed by ebextensions.

With that info and the possibility of concat configuration with the -c option give by Ruflin I could do the magic.

Solution

The solution consist in four steps:

• In the ebextensions I create a simple line comand that generate the yml with the beat name (one per beat)
  echo name: \"filebeat-$HOSTNAME\" > /var/local/filebeat_hostname.yml

• Create a Volumen with that file in the host machine by Dockerun.aws.json
  { "name": "filebeat_hostname", "host": { "sourcePath": "/var/local/filebeat_hostname.yml" } }

• Share this volumen with the docker contaner that run the beat
  { "sourceVolume": "filebeat_hostname", "containerPath": "/var/local/filebeat_hostname.yml" }

• In the configuration of the beat with supervisor add this file with the -c param
  [program:filebeat] command = /opt/filebeat/filebeat-5.0.0-linux-x86_64/filebeat -c /opt/filebeat/filebeat-5.0.0-linux-x86_64/filebeat.yml -c /var/local/filebeat_hostname.yml -v autostart = true autorestart = true priority = 2 stdout_logfile = /var/log/supervisor/filebeat_out.log stderr_logfile = /var/log/supervisor/filebeat_err.log

Conclusion

Mount the beats inside of container is a little headache and maybe for that is not an official image, but is possible and the developers of elastic rocks for make this possible and for give advices for make it in a easy way.

This is not a real problem, all works fine with a simple configuration, the only problem here is that is difficult to find the source of the data. and the main reason of that is the inflexibility of eb to set dynamics environments variables inside of the docker, and not from the beats.

Final note

I know that is a very specific problem, but if this solve future headaches I will be happy

Thanks to @ruflin and @steffens for the help

PS: sorry if something is bad write, I'm doing this as fast as possible

[ruflin]

@lendoly Glad you found a solution and thanks for sharing it with the community.