# Can the "\_source" fields be dropped from events published by filebeat to ES

**URL:** <https://discuss.elastic.co/t/can-the-source-fields-be-dropped-from-events-published-by-filebeat-to-es/205476>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 28, 2019, 1:27pm UTC](https://discuss.elastic.co/t/can-the-source-fields-be-dropped-from-events-published-by-filebeat-to-es/205476 "2019-10-28T13:27:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![john\_eapen](https://avatars.discourse-cdn.com/v4/letter/j/b3f665/32.png) [@john\_eapen](https://discuss.elastic.co/u/john_eapen)\
**Post date:** [October 28, 2019, 1:27pm UTC](https://discuss.elastic.co/t/can-the-source-fields-be-dropped-from-events-published-by-filebeat-to-es/205476/1 "2019-10-28T13:27:17Z")

</div>

Hi

I am using "add\_docker\_metadata" processor.

And when I looked at events published to ES, I see most of the docker metadata common or duplicate under "\_source" as well as "docker" fields.  
Is there a way to eliminate one of these sets. They seem redundant.

"\_source" : {  
"container" : {  
"image" : {  
"name" : "sha256:922c269cf957ec17d66cb5acdfbdd45d420568ec8a6ccd23c6e62b7a7aacb7c8"  
},  
"name" : "k8s\_xxxwin0\_darwin0-6b4f97655-fmdm4\_default\_67d93d84-f5bb-11e9-bf32-00505690ff20\_1",  
"id" : "7f9b94c2b590c04ba123bc487a9a5893060484c9bead76db3c57fca3dde5a083",  
"labels" : {  
"annotation\_io\_kubernetes\_pod\_terminationGracePeriod" : "30",  
...

"docker" : {  
"container" : {  
"labels" : {  
"annotation\_io\_kubernetes\_pod\_terminationGracePeriod" : "30",  
"io\_kubernetes\_container\_logpath" : "/var/log/pods/67d93d84-f5bb-11e9-bf32-00505690ff20/xxxwin0/1.log",  
....

Appreciate any help.  
Thx

---

<div class="post-metadata">

**Author:** ![B.M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/b.m/32/56771_2.png) [@B.M](https://discuss.elastic.co/u/B.M)\
**Post date:** [October 28, 2019, 1:31pm UTC](https://discuss.elastic.co/t/can-the-source-fields-be-dropped-from-events-published-by-filebeat-to-es/205476/2 "2019-10-28T13:31:13Z")

</div>

Hi John,

Could you share your config file?

---

<div class="post-metadata">

**Author:** ![john\_eapen](https://avatars.discourse-cdn.com/v4/letter/j/b3f665/32.png) [@john\_eapen](https://discuss.elastic.co/u/john_eapen)\
**Post date:** [October 28, 2019, 3:53pm UTC](https://discuss.elastic.co/t/can-the-source-fields-be-dropped-from-events-published-by-filebeat-to-es/205476/3 "2019-10-28T15:53:08Z")

</div>

## Hi Badre Thanks for your reply. Here is most of my filebeat config file. ( Feel free to suggest other improvements if you see some. I am still new to filebeat )

apiVersion: v1  
kind: ConfigMap  
metadata:  
name: filebeat-config  
namespace: default  
labels:  
k8s-app: xxx  
app: xxx  
data:

filebeat.yml: |-

```
processors:
- add_cloud_metadata:
- add_host_metadata:
- add_docker_metadata:
    labels.dedot: true
    annotations.dedot: true
- drop_fields:
      #fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "host.name", "ecs.version", "input.type"]
      fields:
        - "agent.ephemeral_id"
        - "agent.hostname"
        - "agent.id"
        - "agent.type"
        - "agent.version"
        - "host.name"
        - "ecs.version"
        - "input.type"
      ignore_missing: true

filebeat.autodiscover:
  providers:
    - type: docker
      # this input section is for module eg: mongo/nginx specific
      templates:
          - condition:
            contains:
              docker.container.name: echo
          config:
            - module: nginx
              enabled: true
              access:
                input:
                  type: container
                  stream: "stdout"
                  containers.ids:
                    - ${data.docker.container.id}
              error:
                input:
                  type: container
                  stream: "stdout"
                  containers.ids:
                    - ${data.docker.container.id}
        - condition:
            contains:
              docker.container.name: darwin
          config:
            - module: mongodb
              enabled: true
              log:
                input:
                  type: docker
                  containers.ids:
                    - ${data.docker.container.id}

filebeat.inputs:
 - type: log
   # this input section is for system log files
   enabled: true
   paths:
    - /var/log/*.log
    - /var/log/messages

 - type: container
   # this input section is for general application docker/containers logs
   enabled: true
   stream: all
   paths:
     - /var/lib/docker/containers/*/*.log
   include_lines: ['"pod":']

   json.keys_under_root: true
   json.ignore_decoding_error: true
   json.add_error_key: true
   json.overwrite_keys: true
   json.message_key: log

  # Avoid parsing exceptions due to name conflicts
   processors:
  - rename:
       fields:
         - from: "service"
           to: "service-name"
         - from: "error"
           to: "service-error"
         - from: "url"
           to: "service-url"
       ignore_missing: true
       fail_on_error: false
```

---

<div class="post-metadata">

**Author:** ![B.M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/b.m/32/56771_2.png) [@B.M](https://discuss.elastic.co/u/B.M)\
**Post date:** [October 28, 2019, 4:32pm UTC](https://discuss.elastic.co/t/can-the-source-fields-be-dropped-from-events-published-by-filebeat-to-es/205476/4 "2019-10-28T16:32:13Z")

</div>

Hi @john_eapen i am sorry but i meant to comment on another post. But i think the sitting you are looking for is [drop\_fields processor](https://www.elastic.co/guide/en/beats/filebeat/master/drop-fields.html), with this setting your can drop other fields as-well except "type" and "@timestamp" since these are default fields to filebeat.  
I hope this answers your question

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 25, 2019, 4:39pm UTC](https://discuss.elastic.co/t/can-the-source-fields-be-dropped-from-events-published-by-filebeat-to-es/205476/5 "2019-11-25T16:39:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.