Filebeat now working on Kubernetes 1.24

[Marco_Lagalla]

Hi,

I have a cluster running Kubernetes version 1.24, hosted on AWS EKS.

Into the cluster there are multiple workloads segregated by namespace.
Filebeat is installed to run as a DaemonSet, and should be able to collect logs for a subset of k8s namespaces.

Till Kubernetes 1.23 all was working fine, but after the removal of Dockershim in Kubernetes 1.24, the old configuration isn't working anymore.

I have tried many different configurations, to collect logs based on containerd:

filebeat:
  bulk_max_size: 8000
filebeat.inputs:
  - type: filestream
    id: obs-filebeat
    paths:
      - /var/log/pods/**/*.log
    prospector.scanner.exclude_files: ['filebeat.*',
                    'logstash.*',
                    'kube.*',
                    'cert-manager.*',
                    'kube-proxy-*', 
                    'fluentd-*', 
                    'cloudwatch-agent-*', 
                    'obs-filebeat-*',
                    'lens-metrics_node-*']
    processors:
      - add_kubernetes_metadata:
          in_cluster: true
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: '/var/log/pods/'
          default_indexers.enabled: false
          default_matchers.enabled: false
          indexers:
            - pod_uid:
          matchers:
            - logs_path:
                logs_path: '/var/log/pods/'
                resource_type: 'pod'
      - drop_event.when.not.and:
        - equals.kubernetes.namespace: "my-app"

processors:
  - add_fields:
      when:
        equals:
          kubernetes.namespace: "my-app"
      target: ''
      fields:
        env: "prod"
        app: "my-app"
  - decode_json_fields:
      fields: ["message","log","logs.log","parsed-message"]
      target: "logs"
      process_array: true
  - drop_fields:
      when:
        regexp:
          message: "^{\""
      fields: ["message"]
      ignore_missing: true
  - drop_fields:
      fields: ["kubernetes.node", "log.file.path","timestamp","std","capital-letter","parsed-message"]
      ignore_missing: true

output.logstash:
  enabled: true
  hosts: ["logstash:5044"]
  bulk_max_size: 8000
  workers: 8
  loadbalance: true
  pipelining: 3
  compression_level: 1
  timeout: 60s
  max_retries: -1
  template.name: filebeat
  template.path: filebeat.template.json

logging.to_files: false

queue.disk:
  path: "${path.data}/persistent-diskqueue"
  max_size: 30GB

When I start Filebeats, logs are collected without any problem and sent to Logstash, but after a while Filebeat seems to stop reading new logs.

{"log.level":"info","@timestamp":"2023-09-05T10:32:52.310Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":8}},"cpuacct":{"total":{"ns":25524289}},"memory":{"mem":{"usage":{"bytes":163725312}}}},"cpu":{"system":{"ticks":1230,"time":{"ms":20}},"total":{"ticks":2220,"time":{"ms":30},"value":2220},"user":{"ticks":990,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"2f6da9c4-2a94-4284-a83b-260249aecef4","uptime":{"ms":360102},"version":"8.8.2"},"memstats":{"gc_next":17970568,"memory_alloc":12250840,"memory_total":162885240,"rss":91398144},"runtime":{"goroutines":27}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":6.33,"15":7.19,"5":7.17,"norm":{"1":1.5825,"15":1.7975,"5":1.7925}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-09-05T10:33:22.309Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":10}},"cpuacct":{"total":{"ns":21054857}},"memory":{"mem":{"usage":{"bytes":163733504}}}},"cpu":{"system":{"ticks":1250,"time":{"ms":20}},"total":{"ticks":2240,"time":{"ms":20},"value":2240},"user":{"ticks":990}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"2f6da9c4-2a94-4284-a83b-260249aecef4","uptime":{"ms":390103},"version":"8.8.2"},"memstats":{"gc_next":17970568,"memory_alloc":13518344,"memory_total":164152744,"rss":91398144},"runtime":{"goroutines":27}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":6.92,"15":7.2,"5":7.22,"norm":{"1":1.73,"15":1.8,"5":1.805}}}},"ecs.version":"1.6.0"}}

Filebeat version: 8.8.2

Any idea?