Filebeat - drop fields processor doesn't remove agent.* and ecs fields. (Without Logstash)

Elastic Stack Beats
1

The drop fields section is working for the other fields like kubernetes.pod.id but it is not working for agent.* and ecs fields.

Any workaround here?

  • Version: 7.3.2
  • Operating System: Linux
  • Steps to Reproduce: Apply the config below.
filebeatConfig:
  filebeat.yml: |
    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition.and:
                - equals.kubernetes.labels.log-group: xxx
              config:
                - type: container
                  paths:
                    - '/var/lib/docker/containers/${data.kubernetes.container.id}/*.log'
                  processors:
                  - drop_fields:
                       fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset"]

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
2

hi @turgayozgur, can you first check if identation is an issue here (I see it is missing before

- drop_fields)

Ex

processors:
 - drop_fields:
     when:
        condition
     fields: ["field1", "field2", ...]
     ignore_missing: false
3

Hi, no difference :confused:

4

Same for me, these fields can't be dropped by Filebeat or Functionbeat when I try with drop_fields. Also version 7.4.0.

agent.ephemeral_id
agent.hostname
agent.id
agent.type
agent.version
ecs.version
host.name
4 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.