I've tried disabling all the processor metadata and somehow narrowed it down but I still can't get rid of agent.ephemeral_id, agent.hostname, agent.id, agent.type, agent.version and ecs.version and log.offset.
Is there a way to disable it or I have to manually specify them in the logstash config to make sure they don't get ingested/indexed ?
You can drop them from filebeat, logstash or configure the mapping/index template to change how they are treated: index or not, doc_values or not, etc.
I'd say you are looking for how to drop fields, from filebeat, you would do it with this:
Thanks @martinr_ubi. I thought there would be an option in filebeat directly to drop all meta data
I'll do it the way you suggested unless someone knows a better way
If this feature exists, it's a hidden undocumented feature so I think it's safe to assume it doesn't exists.
Make sense, those fields are part of the basic functionality of the Beats. You'll break a bunch of stuff by dropping them. But I have to assume you have a custom setup already and are not using any of the features that use those fields.
Ok I tried dropping it from filebeat but it doesn't seem to work anyone has an idea as to way this config fails:
fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset"]
Ok in case this can help some people here it worked it was just some indent issues in the filebeat.yml file.
It works with the following processor definition:
fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset", "version"]
All of it looks like crap to any reading the forums hehe
Always enclose the config you post in proper formatting:
The </> button in the forum post editor or triple back ticks. ```
I prefer the backticks, easier I think.
There is a post preview feature to check what it looks like and it is the only way to share config snippets or files correctly embedded. Or a github gist, etc.
thanks I've edited the previous post with proper formatting in case someone faces the same issue
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.