I was running filebeat 7.10.1 on EKS 1.18 and everything was fine, after the EKS upgrade of EKS to 1.19 filebeat container cannot mount its volume:
MountVolume.SetUp failed for volume "docker-sock" : hostPath type check failed: /var/run/docker.sock is not a file
I am using the following config:
srw-rw---- 1 root docker 0 Apr 26 11:37 /var/run/docker.sock
sudo stat /var/run/docker.sock
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: 17h/23d Inode: 11814 Links: 1
Access: (0660/srw-rw----) Uid: ( 0/ root) Gid: ( 1950/ docker)
Access: 2021-04-26 14:37:37.449086494 +0000
Modify: 2021-04-26 11:37:18.567999944 +0000
Change: 2021-04-26 11:37:18.567999944 +0000
sudo stat /var/lib/docker
Size: 182 Blocks: 0 IO Block: 4096 directory
Device: 10301h/66305d Inode: 273954 Links: 14
Access: (0711/drwx--x--x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-10-12 18:52:05.000000000 +0000
Modify: 2021-04-26 11:37:30.910837483 +0000
Change: 2021-04-26 11:37:30.910837483 +0000
According to this AWS docs the docker version between the latest 1.18 ami and 1.19 ami did not change, there was a change in the kernel used though.
Can you verify if there is any reference to this mount path in the manifest you are using? This is weird because I cannot find any reference in the upstream for filebeat in past versions. We had the same issue in the past with Metricbeat but we removed the specific mount path since it's not useful to be enabled by default: [Kubernetes] Remove redundant dockersock volume mount by ChrsMark · Pull Request #22009 · elastic/beats · GitHub
indeed I do have such a volume and volumeMount in the manifest that is currently rolled out. A colleague of mine snuck this w/o me noticing for unknown reason at the moment.
- name: docker-sock
- name: docker-sock
I will probably remove that once clarified with my colleague but I wonder what changed between the versions that caused the same manifest to error out and prevent the volume from being mounted. (maybe this is not a question for this platform but I still wonder)
Yeap I wonder too cause I think it was only present in Metricbeat's manifests (see my previous comment) but I couldn't spot it in Filebeats' with a quick search in past versions.
To clarify it more, the usage of this socket is required only if someone aim to use
add_docker_metadata processor or want to use
docker metricbeat module. But no need to use this in k8s since we retrieve this info already from other apis/resources.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.