Filebeat does not collect all logs after close.on_state_change.inactive time have passed

Elastic Stack Beats
1

Hello,
Our filebeat (9.0.3) is configured on kubernetes cluster as daemonset to collect logs from applications and to send it to logstash and we have a following configuration:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false

filebeat.inputs:
  - type: filestream
    id: "filebeat-${NODE_NAME}"
    prospector.scanner.symlinks: true
    close.on_state_change.inactive: 10m
    format: cri
    paths:
      - /var/log/containers/*.log
    enabled: true
    parsers:
      - container: ~
    fields:
      logstashSource: "filebeat-k8s--console-json"
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          in_cluster: true
          default_matchers.enabled: false
          matchers:
          - logs_path:
              logs_path: /var/log/containers/
      - drop_event:
          when:
            not.equals.kubernetes.labels.logging-type: "console-json"
      - decode_json_fields:
          fields: ["message"]
          process_array: true
          max_depth: 1
          target: "logData"
      - rename:
          when:
            not.has_fields: ['logData.appName']
          fields:
            - from: "kubernetes.labels.app"
              to: "logData.appName"
      - rename:
          when:
            not.has_fields: ['logData.message']
          fields:
            - from: "message"
              to: "logData.message"
      - rename:
          fields:
            - from: "kubernetes.node.name"
              to: "logData.node"
      - rename:
          fields:
            - from: "fields.logstashSource"
              to: "logData.logstashSource"
      - drop_fields:
          fields: ["stream", "message", "prospector", "offset", "input", "source", "kubernetes", "fields", "log"]

At the beginning everything works as expected and filebeat is collecting all logs (application produces small, single logs). If there is a longer break and last log occurance is greater than 10 minutes (which is close.on_state_change.inactive) we notice a strange behaviour. It looks like filebeat is not processing first few logs. After that next logs are processed correctly. Unfortunatelly those first few logs are missing and never published to Logstash and OpenSearch. I looked at filebeat registry and offset is updated and bigger then before break. Do you have any idea what can be a reason?

Example:
10:00:00 log occures
10:11:00 new log(s) occures (however never published to Logstash)
10:11:15 all next logs are published to Logstash

2

Hi @cubba,

I could imagine a log rotation happening and the logs you're missing being in the active log file while it haven't reached the minimum size for fingerprinting, 1024 bytes. Then the logs rotates, the rotated log file does not match the paths pattern, thus the file is ignored by filebeat, or even if it matches, it does not have the minimum size for fingerprinting and filebeat does not read it.

Do you see any errors or warnings in the logs? Filebeat will log the file is too small to fingerprint and won't be read.

That's what I mean by "log rotation" happening:
- 10:00:00: active log file a.log
- 10:01:00: log rotates: a.log copied to a.log.1, a.log truncated
- 10:01:01 - 10:11:09: a few logs, a.log smaller than 1024 bytes
- 10:11:10: log rotates again
- 10:11:15: several new logs are generated, a.log larger than 1024 bytes

In this scenario, the active log file a.log would not be ingested while it's too small. And if it's rotated before it reaches the minimum size, the rotated file will be ignored or never ingested because it's too small.

If it's the case, you can have a look at the fingerprint options here.

If the problem is indeed the size of the files, a few possible solutions come to mind:

  • configure k8s log rotation to only rotate logs after they're big enough
  • reduce the fingerprint length, but be aware a too small fingerprint length might lead to collisions and therefore data loss
  • consider using a paths pattern that include the rotated log files. But again, they'd need to be bigger than the minimum size for fingerprinting anyway.
1 Like