Remove unnecessary exported fields

[s.buksa (Sintija Bukša)]

Hello,
Is there any way how to remove unnecessary exported fields before indexing?
Using Elastic Agent integration, and along with it comes many exported fields. Tried Elastic Agent processor "drop_fields", but all listed fields for dropping still apears in document. Another option would be to use ingestion pipeline, but at this point it would be more costly and processing in the Agent itself is more preffered.

Just a short example used in configuration file:

processors:
  - drop_fields:
      fields:
        - "cloud.provider"
        - "cloud.region"
        - "host.mac"
        - "host.os.family"
        - "host.os.kernel"
        - "host.os.name"
        - "host.os.codename"
     ignore missing: true

[stephenb (Stephen Brown)]

Hi @s.buksa

What version are you using?

If I recall, unfortunately most of those fields, the host, and agent fields are actually added after The agent processors are run.

I agree, not ideal. So I am fairly sure you're going to have to drop them in an ingest pipeline after they arrive at Elasticsearch.

I'm not sure if the fix of this is on our roadmap, but you certainly can open an issue if you'd like.

[s.buksa (Sintija Bukša)]

Hi,
Thank you for your reply. I'm using version 8.9.2.