Filebeat does not mount volume

Hi,
I was running filebeat 7.10.1 on EKS 1.18 and everything was fine, after the EKS upgrade of EKS to 1.19 filebeat container cannot mount its volume:

MountVolume.SetUp failed for volume "docker-sock" : hostPath type check failed: /var/run/docker.sock is not a file

I am using the following config:

https://raw.githubusercontent.com/elastic/beats/7.10/deploy/kubernetes/filebeat-kubernetes.yaml
ll /var/run/docker.sock
srw-rw---- 1 root docker 0 Apr 26 11:37 /var/run/docker.sock



sudo stat /var/run/docker.sock
  File: ‘/var/run/docker.sock’
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 17h/23d Inode: 11814       Links: 1
Access: (0660/srw-rw----)  Uid: (    0/    root)   Gid: ( 1950/  docker)
Access: 2021-04-26 14:37:37.449086494 +0000
Modify: 2021-04-26 11:37:18.567999944 +0000
Change: 2021-04-26 11:37:18.567999944 +0000
 Birth: -



sudo stat /var/lib/docker
  File: ‘/var/lib/docker’
  Size: 182             Blocks: 0          IO Block: 4096   directory
Device: 10301h/66305d   Inode: 273954      Links: 14
Access: (0711/drwx--x--x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-10-12 18:52:05.000000000 +0000
Modify: 2021-04-26 11:37:30.910837483 +0000
Change: 2021-04-26 11:37:30.910837483 +0000
 Birth: -

According to this AWS docs the docker version between the latest 1.18 ami and 1.19 ami did not change, there was a change in the kernel used though.

https://docs.aws.amazon.com/eks/latest/userguide/eks-linux-ami-versions.html

Hi!

Can you verify if there is any reference to this mount path in the manifest you are using? This is weird because I cannot find any reference in the upstream for filebeat in past versions. We had the same issue in the past with Metricbeat but we removed the specific mount path since it's not useful to be enabled by default: [Kubernetes] Remove redundant dockersock volume mount by ChrsMark · Pull Request #22009 · elastic/beats · GitHub

Hi Chris,
indeed I do have such a volume and volumeMount in the manifest that is currently rolled out. A colleague of mine snuck this w/o me noticing for unknown reason at the moment.

.....
volumeMounts:
        - name: docker-sock
          mountPath: /var/run/docker.sock
          readOnly: true
...
volumes:
- name: docker-sock
  hostPath:
    path: /var/run/docker.sock
    type: FileOrCreate

I will probably remove that once clarified with my colleague but I wonder what changed between the versions that caused the same manifest to error out and prevent the volume from being mounted. (maybe this is not a question for this platform but I still wonder)

Regards,
Alex

Yeap I wonder too :slightly_smiling_face: cause I think it was only present in Metricbeat's manifests (see my previous comment) but I couldn't spot it in Filebeats' with a quick search in past versions.

To clarify it more, the usage of this socket is required only if someone aim to use add_docker_metadata processor or want to use docker metricbeat module. But no need to use this in k8s since we retrieve this info already from other apis/resources.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.