I'm using filebeat inside of kubernetes both as daemonset (to capture stdout of all containers), but also as sidecar in specific pods, to ship application logs that are written to shared volumes. The documentation is somewhat lacking about the required steps/gotchas, especially in recent versions.
To note:
- recent versions of filebeat-kubernetes.yaml include some roles and cluster roles that are apparently required by the autodiscover, but are not explained or documented anywhere. Not sure why the leases role is supposed to be required for, for instance. I seem to remember digging through the source code and coming to the conclusion it was for a dropped feature. I remove it in my setup with no visible downside.
- in recent ELK releases, agent monitoring is encouraged to be done via metricbeat, but that's not really an option when running filebeat as a container. I don't see neither the daemonset nor the sidecar instances in the stack monitoring page in kibana, not sure what the required configuration would be
- sidecar filebeat is a popular solution to ship logs, however the official documentation does not mention it, and it has some gotchas that would be nice to see in the docs: how to populate pod metadata? does it require giving the whole pod additional privileges? how to set beat name for monitoring? other useful processors that would be recommended to consider?