Filebeat - How to tag multiple log files in kubernetes pod?

[sivakr (Siva Kumar)]

I have a pod which writes multiple log files (in different format) like

1. /var/log/custom1.log
2. /var/log/custom2.log

And the pod has label: payment

Now how do I create a Filebeat DeamonSet which looks for pod with label: payment and ships the custom1.log with tag format_1 and custom2.log with tag format_2 to Logstash

And since pod's are ephemeral it should ship logs even if new pods are added.

It would be really helpful if I get a Filebeat configuration for the above requirement.

Thanks

[steffens (Steffen Siering)]

You can find kubernetes deployment samples in the beats repository.

For filebeat custom modules and inputs can be configured by adding your own input configurations to the filebeat inputs.d subdirectory (moundPath: /usr/share/filebeat/inputs.d).

Question is: where exactly does the pod write these files too? Are you using shared volumes? As filebeat is running as a DaemonSet it needs to know the locations to pick up the files from + the location must be mounted into the filebeat container.

For example the ConfigMap configures the docker log collector. For the input to work, the /var/lib/docker/containers directory is mounted by the DaemonSet.

Using Autodiscovery, you can use container labels to configure collection and additional meta data to be added to the collected events. The docs contain some samples.