Beat File Output Dynamic Naming

Elastic Stack Beats
1

Hi all,

I am using the File Output functionality of Metricbeat and Winlogbeat 5.5.1, and I'd like the ability to add beat data to the output filename. Specifically the date component of the @timestamp field. Does the Beat file output filename field support beat data retrieval? If so, what is the syntax?

I've attempted syntax such as:
filename: "metricbeat-%{{beat.hostname}}-%{{@timestamp}}"
filename: "metricbeat-{{beat.hostname}}-{{@timestamp}}"
filename: "metricbeat-{[beat.hostname]}-{[@timestamp]}"

Quick back story- I'm attempting to create a monitoring solution of my Production environment, with the constraint that my ES instance is in a different domain, with no direct connectivity (gotta love corporate IT...). The only connectivity between those environments is a network share acting as a dropbox. Due to this, I have file output configured on my beats in Prod, and I'd like to make those outputted filenames unique so they aren't overwriting each other when I moved them from Prod into my ES environment.

1 Like
2

The file output doesn't support using format strings.

3

Got it, thanks for the quick reply Andrew!

4

Ouch.

As beats are kind of a streaming producers, the problem with this file naming scheme is, when do we have to open/reopen/close those files in a sane manner.

Some potential workarounds:

Do you have at some automation (speak scripts) downloading the actual log files? As file output by default is one json document per line, the script can parse the initial line and extract the initial offset from that file.

Or have a cronjob restarting the beats every 24h. A beats configuration file can be split into multiple files. Plus one can pass additional settings via -E flag on command line.
E.g. update the init script to read filebeat ... -E output.file.filename="<beatname>-$(hostname)-$(date +%Y-%m-%d). After having updated the init script just add a service restart to your daily cron.
This is some change/adaptation one can easily integrate with ansible/puppet/chef/...

5

This topic was automatically closed after 21 days. New replies are no longer allowed.