Removing agent.* metadata doesn't work in Filebet nor Logstash

[ehsank777 (Ehsan)]

Hi,

I have Elasticsearch and Logstash version 7.5.0 installed in an ELK stack. So far I was using filebeat version 6.8.0 on my machines. Today I upgraded some of them to version 7.6.0 and after that I see the following error on logstash logs:

[2020-03-05T15:16:38,462][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x4ad6adbe], :response=>{"index"=>{"_index"=>"logstash-000060", "_type"=>"_doc", "_id"=>"iqAOq3ABYeru0YScLwc4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [agent] of type [text] in document with id 'iqAOq3ABYeru0YScLwc4'. Preview of field's value: '{hostname=mymachine.domain, id=338d09d1-3789-4053-b85b-67be12fad7b0, ephemeral_id=8c208fc4-a28a-460f-94cd-2154ef5ccce4, type=filebeat, version=7.6.0}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:156"}}}}}

And the cause is that I have another text field on my template called "agent" and I cannot change it.
So , I wanted to remove all agent.* metadata in filebeat or logstash.
I used https://discuss.elastic.co/t/solved-how-to-remove-agent-and-ecs-version/183643/9 for filebeat and the following for logstash

 if [agent.type] {
  mutate {
        add_field => [ "beat.hostname", "%{agent.hostname}" ]
        add_field => [ "beat.version" , "%{agent.version}"  ]
       remove_field => [ "agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version" ]
  }
  }

But unfortunately none of them worked, and I still recive the same error log.
How can I remove them?

[dfinn (Daniel J Finn)]

I'm dealing with some similar issues. Here's what I did in my filebeat.yml config:

  - rename:
      fields:
       - from: "agent"
         to: "filebeat_agent"

That fixed that issue but now I'm still having a problem with elasticsearch expecting these agent fields on my daily indexes and it's causing problems with my pipeline every morning.

[ehsank777 (Ehsan)]

Thanks @dfinn your solution could help at the end, but since I have many machines with filebeat configured, it'd be tedious. I'd rather remove them on logstash side.

[dfinn (Daniel J Finn)]

Gotcha. We use puppet to manage the filebeat config so it was easy for me to make this change on multiple servers.

[ehsank777 (Ehsan)]

I finally got logstash to work for removing these fields, the following does work:

 if [agent][type] {
  mutate {
        add_field => [ "beat.hostname", "%{[agent][hostname]" ]
        add_field => [ "beat.version" , "%{[agent][version]"  ]
       remove_field => [ "[agent][ephemeral_id]", "[agent][hostname]",  ......]
  }
  }