Filebeat stop working after upgrade from 7.6.0 to 7.7.1 (kubernetes)

Elastic Stack Beats
1

I have filebeat running as daemonset in k8s cluster, while apps storing file which mounted in host disk, this path also mounted by filebeat containers.
Previously on 7.6 its working but after upgrading to 7.7.1 its stop. Rolling back to 7.6.0 make it working again.
after upgrading to 7.7.1, the only obvious error in Filebeat container is:

ERROR	[kubernetes]	add_kubernetes_metadata/matchers.go:91	Error extracting container id - source value does not contain matcher's logs_path '/var/lib/docker/containers/'.

This error not occur on filebeat 7.6.0

this is my filebeat yaml:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.pod.labels.logtype: filebeat
              config:
                - type: log
                  enabled: true
                  paths:
                    - /var/lib/kubelet/pods/${data.kubernetes.pod.uid}/volumes/kubernetes.io~empty-dir/filebeat-volume/*.json
                  json:
                    keys_under_root: false
                    add_error_key: true
                  fields:
                    logtype: filebeat

while filebeat mounting /var/lib/kubelet/pods/ and my pods have labels logtype=filebeat
Any help is appreciated

2

See: Problem to update to filebeat 7.7.0 and parser nginx-ingress-controller on Kubernetes

3

I already read it, but it didn't explain why it worked in 7.6.0 but not in 7.7.1
Furthermore the java apps log is already pick up by filebeat (/var/log/containers/), but since the other log (/var/lib/kubelet/pods/) is for json logs, we need to change our code to combine both and print it on stdout (/var/log/containers/) to make it work with 7.7.1?

Note: /var/lib/kubelet/pods/ is the default directory of kubernetes to store emptyDir volumes, and we use in to put all json logs, so diminished the need of logstash

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.