I'm trying to perform node level logging in a kubernetes cluster using Filebeat. I've installed Filebeat as a daemonset and it is reading logs from /var/lib/docker/containers//*.log successfully. My concern is that I am having to run the Filebeat container as root because root owns /var/lib/docker/containers and the permissions for the directory are set to 700. I would like to avoid running containers as root for security concerns. I've looked at modifying the /var/lib/docker/containers permissions using an initcontainer, but that does not address any pods mounted after the Filebeat initcontainer runs. I looked at reading the logs through /var/log/pods/... but that would require using symbolic links and your docs recommend against relying on symbolic links.
What is considered best practice in a production environment:
- avoiding node level logging altogether and just relying on sidecars for logging
- running the filebeat container in a daemonset as root as I am doing
- relying on symbolic links and managing those challenges
Or, is there an alternate approach that I am overlooking?