Filebeat processors drop_fields has no effect

Elastic Stack Beats
1

background
this processors can not drop field "agent_name"
image

my filebeat config

filebeat.inputs:
- type: container
  paths:
    # - /var/log/containers/xgimi-launcher*.log
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
    - drop_fields:
        fields: ["kubernetes_labels_app"]

  multiline.type: pattern
  multiline.pattern: '.*exception.*:.*|^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      node: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - timestamp:
      field: "@timestamp"
      layouts:
        - '2006-01-02T15:04:05Z'
        - '2006-01-02T15:04:05.999Z'
        - '2006-01-02T15:04:05.999-07:00'
  - drop_fields:
      fields: ["agent_name"]


cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}

output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["10.64.99.29:9092", "10.64.99.2:9092", "10.64.99.238:9092"]

  # message topic selection + partitioning
  topic: 'filebeat-chen'
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
2

Hello and welcome,

What is your final output, after kafka, the source of your screenshot?

Filebeat does not produce fields with underscore, it will produce a json object named agent with multiple nested fields, like name, version, id etc.

So you will have agent.name, agent.version, agent.id etc.

Change agent_name in your drop_fields processor to agent.name.

3

thks, this message was reprocessed by graylog extractors.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.