Filebeat mixing pod logs

[mnbill]

Hello

May be anyone can check that ?

I am facing mixing logs on windows filebeat.
I build a windows filebeat docker image (based on windows server 1909) and i deployed daemonset on k8s cluster, everyting work fine with v7.10.2, after upgrading to filebeat 7.16.2, i see mixing logs.

I built a windows filebeat docker image (based on windows server 1909) with this Dockefile:

FROM mcr.microsoft.com/windows/servercore:1909

LABEL Name="filebeat-win" \
description="Base Windows servercore 1909 Image"

RUN mkdir C:\\app 

ADD https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.16.2-windows-x86_64.zip  C:\\app 

WORKDIR C:\\app

RUN powershell -Command  \
    Expand-Archive -Path 'c:\app\filebeat-7.16.2-windows-x86_64.zip'; \ 
    Move-Item -Path  'c:\app\filebeat-7.16.2-windows-x86_64\filebeat-7.16.2-windows-x86_64\*' -Destination 'c:\app\'; \
    Remove-Item 'c:\app\filebeat-7.16.2-windows-x86_64.zip' -ErrorAction Ignore
    

RUN mkdir C:\\etc\\filebeat

COPY filebeat.yml C:\\etc\\filebeat

ENTRYPOINT ["c:\\app\\filebeat.exe", "-c", "c:\\etc\\filebeat\\filebeat.yml", "-e"]

I'm using daemonset of windows filebeat 7.16.2 in k8s cluster with config:

    filebeat.autodiscover:
      providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints:
          enabled: true
          default_config:
            type: container
            paths:
              - "C:\\var\\log\\containers\\*"

    - add_kubernetes_metadata:
        default_matchers.enable: false
        matchers:
          - logs_path:
              logs_path: 'C:\var\log\containers\'

On filebeat log is see that some containers are poiting to another log.file.path (of other container), on filebeat log:

2022-01-18T20:10:23.322+0100    DEBUG   [processors]    processing/processors.go:203    Publish event: {
  "@timestamp": "2022-01-18T19:10:23.276Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.16.2"
  },
  "ecs": {
    "version": "1.12.0"
  },
  "host": {
    "name": "filebeat-win1909-gjr2w"
  },
  "agent": {
    "type": "filebeat",
    "version": "7.16.2",
    "hostname": "filebeat-win1909-gjr2w",
    "name": "filebeat-win1909-gjr2w"
  },
  "hosting": {
    "name": "k8s",
    "type": "k8s"
  },
  "log": {
    "file": {
      "path": "C:\\var\\log\\containers\\filebeat-win1909-gjr2w_filebeat-ns_filebeat-win1909-d61d412f80a9ffb00816657e24d718eb569cba8dfc5f36de5b5f343717f3f1ce.log"
    },
    "offset": 56411
  },
  "stream": "stderr",
  "input": {
    "type": "container"
  },
  "container": {
    "id": "6fb3c1c177485975b57489bfd6328c501158736d81c6c06c08f71257ec5b787f",
    "runtime": "docker",
    "image": {
      "name": "iis-log:v1"
    }
  },
  message": "2022-01-18T20:10:23.276+0100\tDEBUG\t[registrar]\tregistrar/registrar.go:263\tProcessing 1 events",
  "kubernetes": {
    "container": {
      "name": "iis-test-kubelet"
    },
    "node": {
      "name": "nodewin"
    }, 
    "pod": {
      "name": "iis-test-kubelet-5f5dcb89b9-6w8nx"
    },
    "namespace": "test-ns",
    "replicaset": {
      "name": "iis-test-kubelet-5f5dcb89b9"
    },
    "labels": {
      "app": "iis-test-kubelet",
    }
  }
}

in this case for the pod iis-test-kubelet is not poiting to the correct log.file.path (is poiting to windows filebeat container log instead of C:\var\log\containers\iis-test-kubelet-XXXX.log .