We have Filebeat running on K8, collecting logs from multiple inputs on 2 namespaces.
I am now working with another team, which has applications also deployed to Kubernetes under a different namespace. We went ahead with the most obvious solution, which is to add filebeat inputs to the filebeat-kubernetes.yaml file.
My concern is we now have a single filebeat-kubernetes.yaml file that is increasing in size. It has 12 inputs and just had a 13th input added, which will surely be broken out into more inputs as the other team figures out the names of their log files being deposited into /var/log/containers/ by K8.
Is there a way to break down the single file into multiple files?
I eventually found a solution to this scenario.
I got another copy of the default filebeat-kubernetes.yaml manifest provided by Elastic here: Run Filebeat on Kubernetes | Filebeat Reference [8.15] | Elastic
If you deploy Filebeat as a DaemonSet with the above file, it will use a ServiceAccount, ClusterRole, ConfigMap, under the default filebeat name.
So if you want to deploy a second Filebeat DaemonSet, you would use another manifest file, and edit ServiceAccount, ClusterRole, ConfigMap, etc. to use a name that is different than the name used by the first DaemonSet.
So if you have say 3 software development teams running 3 different projects that are all deployed to the same Kubernetes node, you could set up 3 different manifest files, instead of configuring one monster size manifest file with settings for all 3 projects.
More details here:
Filebeat DaemonSet on Kubernetes not shipping logs - Elastic Stack / Beats - Discuss the Elastic Stack
I have to post a correction here.
I had thought that creating a 2nd Filebeat manifest file, as described above, then running kubectl create -f on it was working.
The past few weeks I have been struggling to figure out why I was only able to get the new application logs indexed into Elasticsearch that one time, and only that one time.
I copied the filestream input lines for the new application to the older, established manifest file, did the kubectul delete and create commands on that one, and now the new application files are being indexed as expected.
So for now I can only conclude that what I thought was a solution was a failure. To get more than one Filebeat instance working in the same K8 cluster is not as simple as copying a manifest file and changing the service name, unfortunately.
For the time being, we have no choice but to use one huge and continually growing manifest file to configure Filebeat to collect logs from 20 different applications or whatever the final number will be.