# Removing agent.\* metadata doesn't work in Filebet nor Logstash

**URL:** <https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314>\
**Category:** Logstash\
**Created:** [March 5, 2020, 2:30pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314 "2020-03-05T14:30:57Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ehsank777](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ehsank777/32/64420_2.png) [@ehsank777](https://discuss.elastic.co/u/ehsank777)\
**Post date:** [March 5, 2020, 2:30pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314/1 "2020-03-05T14:30:57Z")

</div>

Hi,

I have Elasticsearch and Logstash version 7.5.0 installed in an ELK stack. So far I was using filebeat version 6.8.0 on my machines. Today I upgraded some of them to version 7.6.0 and after that I see the following error on logstash logs:

> [2020-03-05T15:16:38,462][WARN][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash", :\_type=\>"\_doc", :routing=\>nil}, #LogStash::Event:0x4ad6adbe], :response=\>{"index"=\>{"\_index"=\>"logstash-000060", "\_type"=\>"\_doc", "\_id"=\>"iqAOq3ABYeru0YScLwc4", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [agent] of type [text] in document with id 'iqAOq3ABYeru0YScLwc4'. Preview of field's value: '{hostname=mymachine.domain, id=338d09d1-3789-4053-b85b-67be12fad7b0, ephemeral\_id=8c208fc4-a28a-460f-94cd-2154ef5ccce4, type=filebeat, version=7.6.0}'", "caused\_by"=\>{"type"=\>"illegal\_state\_exception", "reason"=\>"Can't get text on a START\_OBJECT at 1:156"}}}}}

And the cause is that I have another text field on my template called "agent" and I cannot change it.  
So , I wanted to remove all agent.\* metadata in filebeat or logstash.  
I used [https://discuss.elastic.co/t/solved-how-to-remove-agent-and-ecs-version/183643/9](https://discuss.elastic.co/t/solved-how-to-remove-agent-and-ecs-version/183643/9) for filebeat and the following for logstash

```
 if [agent.type] {
  mutate {
        add_field => ["beat.hostname", "%{agent.hostname}"]
        add_field => ["beat.version" , "%{agent.version}"]
       remove_field => ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version"]
  }
  }

```

But unfortunately none of them worked, and I still recive the same error log.  
How can I remove them?

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [March 5, 2020, 4:48pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314/2 "2020-03-05T16:48:53Z")

</div>

I'm dealing with some similar issues. Here's what I did in my filebeat.yml config:

```auto
  - rename:
      fields:
       - from: "agent"
         to: "filebeat_agent"

```

That fixed that issue but now I'm still having a problem with elasticsearch expecting these agent fields on my daily indexes and it's causing problems with my pipeline every morning.

---

<div class="post-metadata">

**Author:** ![ehsank777](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ehsank777/32/64420_2.png) [@ehsank777](https://discuss.elastic.co/u/ehsank777)\
**Post date:** [March 5, 2020, 5:19pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314/3 "2020-03-05T17:19:26Z")

</div>

Thanks @dfinn your solution could help at the end, but since I have many machines with filebeat configured, it'd be tedious. I'd rather remove them on logstash side.

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [March 5, 2020, 5:21pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314/4 "2020-03-05T17:21:52Z")

</div>

Gotcha. We use puppet to manage the filebeat config so it was easy for me to make this change on multiple servers.

---

<div class="post-metadata">

**Author:** ![ehsank777](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ehsank777/32/64420_2.png) [@ehsank777](https://discuss.elastic.co/u/ehsank777)\
**Post date:** [March 9, 2020, 2:27pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314/5 "2020-03-09T14:27:31Z")

</div>

I finally got logstash to work for removing these fields, the following does work:

```
 if [agent][type] {
  mutate {
        add_field => ["beat.hostname", "%{[agent][hostname]" ]
        add_field => ["beat.version" , "%{[agent][version]" ]
       remove_field => ["[agent][ephemeral_id]", "[agent][hostname]", ......]
  }
  }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2020, 2:27pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314/6 "2020-04-06T14:27:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.