Trying to understand Filebeat, Kubernetes logrotation and missing logs

Elastic Stack Beats
1

Hi we are running Filbeat 8.9.2 as a Daemonset in Kubernetes. Filebeat is configure to insert Kafka. Here is the config:

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"

processors:
  - if:
      equals:
        kubernetes.labels.log-group: raange
    then:
      add_fields:
        fields:
          log-topic: app-logs-kube
    else:
      add_fields:
        fields:
          log-topic: kube-other

output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["xxxxxx.xxxxxx:9092", "xxxxxx.xxxxxx:9092", "xxxxxx.xxxxxx:9092"]

  # message topic selection + partitioning
  topic: '%{[fields.log-topic]}'
  partition.round_robin:
    reachable_only: true

  required_acks: -1
  max_message_bytes: 2000000

Also Filebeat Daemon set is configured to request 100m cpu request 100Mi meme and limit of 200Mi mem.

I have Kubelete to rotate log every 100MB and keep 20 files based on the setting specified here: Logging Architecture | Kubernetes (This seems to hint that Kubelete is doing the rotation and not logrotate, more on this later)

  1. In /var/log/pods/* I do indeed see logs being rotated every 100MB.
  2. I also see one file of 100MB being kept around.
  3. The extra file are being gziped (More on this later as well)
  4. It keeps 20 GZ files.

So Based on the above Filebeat configs.

  1. On peek time we can do about 3000-4000 logs per seconds from all pods, some pods hitting 600-800 logs per second. Are the CPU request and mem limits enough?
  2. I see the default Filebeat Daemonset config we download point to /var/log/containers/* which is bassically symlinks to /va/log/pods/* I'm assuming filebeat and the filesystem are smart enough to know when the rotation happens that the pointer changes.
  3. Based on above as noted Kubelete seems to keep the active log ,1 file uncompressed, and the rest compressed. While rotation happens is it possible at that hight volume we lose logs (I'm not talking about 3-4 logs until rotation happens but thousands, like 100K+)?
  4. Given filebeat read the symlink, is filebeat/filesystem smart enough to keep reading the left over uncompressed file and the active at same time, or we have to specifically point to var/log/pods/*
  5. Is it better to point to /var/log/pods/* and does the kubernetes detection still work with that folder?

Regarding logrotate, alot of resources on the internet seem to point and say that Kubernetes uses logrotate for the rotation mechanism. I have looked on all the Kubernetes agents and I have found nothing that indicates so.

In /etc/logrotate.d/ I see the following folders alternatives apport apt bootlog btmp dpkg netdata rsyslog ubuntu-advantage-tools ufw unattended-upgrades wtmp. I alsp physically looked inside and nothing pointed to var/log/containes/ or var/log/pods/.

In /etc/logrotate.conf we have:

weekly
su root adm
rotate 4
create
include /etc/logrotate.d

Which doesn't correspond to any of the logrotation behavior. I.e: Rotate every 100MB, compress and keep 20 files.

The docs Logging Architecture | Kubernetes seem to indicate that Kubelet is doing the rotation, but no mention of compression anywhere? or is Kubelet configuring log rotate somewhere?