[SOLVED]How to remove agent.* and ecs.version?

[Duked]

Hi,

I've tried disabling all the processor metadata and somehow narrowed it down but I still can't get rid of agent.ephemeral_id, agent.hostname, agent.id, agent.type, agent.version and ecs.version and log.offset.

Is there a way to disable it or I have to manually specify them in the logstash config to make sure they don't get ingested/indexed ?

thanks !

[martinr_ubi (Martin R.)]

You can drop them from filebeat, logstash or configure the mapping/index template to change how they are treated: index or not, doc_values or not, etc.

I'd say you are looking for how to drop fields, from filebeat, you would do it with this:
https://www.elastic.co/guide/en/beats/filebeat/current/drop-fields.html

[Duked]

Thanks @martinr_ubi. I thought there would be an option in filebeat directly to drop all meta data

I'll do it the way you suggested unless someone knows a better way

[martinr_ubi (Martin R.)]

If this feature exists, it's a hidden undocumented feature so I think it's safe to assume it doesn't exists.
Make sense, those fields are part of the basic functionality of the Beats. You'll break a bunch of stuff by dropping them. But I have to assume you have a custom setup already and are not using any of the features that use those fields.

[Duked]

Ok I tried dropping it from filebeat but it doesn't seem to work anyone has an idea as to way this config fails:

processors:
- drop_fields:
when:
equals
fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset"]

from filebeat.yml

[Duked]

Ok in case this can help some people here it worked it was just some indent issues in the filebeat.yml file.

It works with the following processor definition:

processors:
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset", "version"]

[martinr_ubi (Martin R.)]

All of it looks like crap to any reading the forums hehe

Always enclose the config you post in proper formatting:
The </> button in the forum post editor or triple back ticks. ```
I prefer the backticks, easier I think.

Like this:

processors:
- <processor_name>:
    when:
      <condition>
    <parameters>

- <processor_name>:
    when:
      <condition>
    <parameters>

There is a post preview feature to check what it looks like and it is the only way to share config snippets or files correctly embedded. Or a github gist, etc.

[Duked]

thanks I've edited the previous post with proper formatting in case someone faces the same issue