Filebeat - processors parameters not being applied to all logs

Elastic Stack Beats
,
1

Fairly new to this. I'm trying to understand why some of our Filebeat logs are not being subject to the parameters defined in "processors". It is my understanding that the processor parameters should be applied to all data collected by Filebeat.

For example, when consulting airflow-log-1, I see the processor parameters (add_fields, add_kubernetes_metadata, add_cloud_metadata, add_host_metadata) have been applied. But when consulting airflow-log-2, all the processor parameters have not been applied. Note that there a difference of milliseconds in the timestamp of both of those logs.

Here's the relevant section of my values' file:

daemonset:
  filebeatConfig:
    filebeat.yml: |
      filebeat.autodiscover:
        providers:
          - type: kubernetes
            templates:
              - config:
                  - type: container
                    paths:
                      - /var/log/containers/*-${data.kubernetes.container.id}.log
                    exclude_lines: ["^\\s+[\\-`('.|_]"]
      filebeat.inputs:
      - type: container
        paths:
          - /etc/beat.yml
          - /usr/share/filebeat/data
          - /var/lib/docker/containers
          - /var/log/containers
          - /var/log/containers/*.log
          - /var/log/pods
          - /var/run/secrets/kubernetes.io/serviceaccount
        processors:
          - add_id: ~
          - add_fields:
              target: "application"
              fields:
                name: "doggo"
                cluster: "doggo-prod"
                environment: "production"
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
              - logs_path:
                  logs_path: "/var/log/containers/"
          - add_cloud_metadata: {}
          - add_host_metadata: {}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.