How to remove fields not required when sending logs via elastic agent

[HHobeck]

Dear community.

I have exactly the same issue like shi in Reference [1] but with the different that I'm using elastic agent with custom log integration. Under the surface I guess file beat will be used but I have no luck with the configuration.

The processors configuration looks like the following (yes it is json because to avoid whitespace hell):

[
  ...
  {
    "drop_fields": {
      "fields": ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset", "version"]
    }
  }
]

I'm still getting all the meta fields in the document on the index e.g. ecs.version:

{
    ...
    "ecs": {
      "version": "8.0.0"
    },
    ...
}

Can anyone help me please?

Regards

HHobeck

Reference [1]: How to remove fields not required while sending log data from file beat
Reference [2]: [SOLVED]How to remove agent.* and ecs.version? - #3 by Duked
Reference [3: Filebeat didn't drop some of the fields like agent.*, ecs.* etc

[HHobeck]

I have wrote a script processor step and print out the event object as JSON. It seems to be that the property ecs.version and others are not present in this object at this stage. I think it will be enriched later. That is the reason why the drop_fields processor step doesn't work properly.

How to avoid enriching the event with such unneeded meta data?

[leandrojmp (Leandro Pereira)]

@HHobeck

How to avoid enriching the event with such unneeded meta data?

You will need to remove those using a custom Ingest Pipeline for your integration.

Also, If you are using Fleet I'm not sure you should remove the agent fields as those fields seems to be used by Fleet, but only someone from Elastic can confirm.