Filebit does not see some of the containers on the host in the kubernetes

[pastukhov (Artem Pastukhov)]

Good day!

Today I found that filebit does not collect logs from a part of the containers in my cluster

My config:

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
      inputs:
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: false
    filebeat.inputs:
    - enabled: true
      paths:
      - /var/log/*.log
      - /var/log/messages
      - /var/log/syslog
      type: log
    - containers.ids:
      - "*"
      multiline:
        match: after
        negate: false
        pattern: '^[[:space:]]|^Caused by:'
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_event:
          when:
            or:
            - equals:
                kubernetes.container.name: filebeat
            - equals:
                kubernetes.container.name: logstash
      type: docker

Filebeat version: 6.7.0

Number of containers harvested:

    kubectl logs filebeat-zw7xr -c filebeat | grep Harvester | grep docker |wc -l
    12

Number of containers started:

    docker ps | wc -l
    45

Number of logfiles:

    find /var/lib/docker/containers -name *json.log | wc -l
    47

Can anyone help?

[exekias (Carlos Pérez Aradros)]

Hi @pastukhov,

docker does detection based on file listing, so I'm wondering what's going on here, could it be that Filebeat doesn't have permission to read these files? You could also enable debug mode, by passing -d '*' param, to see more details.

Best regards