Removing agent.* metadata doesn't work in Filebet nor Logstash


I have Elasticsearch and Logstash version 7.5.0 installed in an ELK stack. So far I was using filebeat version 6.8.0 on my machines. Today I upgraded some of them to version 7.6.0 and after that I see the following error on logstash logs:

[2020-03-05T15:16:38,462][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x4ad6adbe], :response=>{"index"=>{"_index"=>"logstash-000060", "_type"=>"_doc", "_id"=>"iqAOq3ABYeru0YScLwc4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [agent] of type [text] in document with id 'iqAOq3ABYeru0YScLwc4'. Preview of field's value: '{hostname=mymachine.domain, id=338d09d1-3789-4053-b85b-67be12fad7b0, ephemeral_id=8c208fc4-a28a-460f-94cd-2154ef5ccce4, type=filebeat, version=7.6.0}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:156"}}}}}

And the cause is that I have another text field on my template called "agent" and I cannot change it.
So , I wanted to remove all agent.* metadata in filebeat or logstash.
I used for filebeat and the following for logstash

 if [agent.type] {
  mutate {
        add_field => [ "beat.hostname", "%{agent.hostname}" ]
        add_field => [ "beat.version" , "%{agent.version}"  ]
       remove_field => [ "agent.ephemeral_id", "agent.hostname", "", "agent.type", "agent.version" ]

But unfortunately none of them worked, and I still recive the same error log.
How can I remove them?

I'm dealing with some similar issues. Here's what I did in my filebeat.yml config:

  - rename:
       - from: "agent"
         to: "filebeat_agent"

That fixed that issue but now I'm still having a problem with elasticsearch expecting these agent fields on my daily indexes and it's causing problems with my pipeline every morning.

1 Like

Thanks @dfinn your solution could help at the end, but since I have many machines with filebeat configured, it'd be tedious. I'd rather remove them on logstash side.

Gotcha. We use puppet to manage the filebeat config so it was easy for me to make this change on multiple servers.

1 Like

I finally got logstash to work for removing these fields, the following does work:

 if [agent][type] {
  mutate {
        add_field => [ "beat.hostname", "%{[agent][hostname]" ]
        add_field => [ "beat.version" , "%{[agent][version]"  ]
       remove_field => [ "[agent][ephemeral_id]", "[agent][hostname]",  ......]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.