Can the "_source" fields be dropped from events published by filebeat to ES

Elastic Stack Beats
1

Hi

I am using "add_docker_metadata" processor.

And when I looked at events published to ES, I see most of the docker metadata common or duplicate under "_source" as well as "docker" fields.
Is there a way to eliminate one of these sets. They seem redundant.

"_source" : {
"container" : {
"image" : {
"name" : "sha256:922c269cf957ec17d66cb5acdfbdd45d420568ec8a6ccd23c6e62b7a7aacb7c8"
},
"name" : "k8s_xxxwin0_darwin0-6b4f97655-fmdm4_default_67d93d84-f5bb-11e9-bf32-00505690ff20_1",
"id" : "7f9b94c2b590c04ba123bc487a9a5893060484c9bead76db3c57fca3dde5a083",
"labels" : {
"annotation_io_kubernetes_pod_terminationGracePeriod" : "30",
...

"docker" : {
"container" : {
"labels" : {
"annotation_io_kubernetes_pod_terminationGracePeriod" : "30",
"io_kubernetes_container_logpath" : "/var/log/pods/67d93d84-f5bb-11e9-bf32-00505690ff20/xxxwin0/1.log",
....

Appreciate any help.
Thx

2

Hi John,

Could you share your config file?

3

Hi Badre
Thanks for your reply. Here is most of my filebeat config file.
( Feel free to suggest other improvements if you see some. I am still new to filebeat )

apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: default
labels:
k8s-app: xxx
app: xxx
data:

filebeat.yml: |-

processors:
- add_cloud_metadata:
- add_host_metadata:
- add_docker_metadata:
    labels.dedot: true
    annotations.dedot: true
- drop_fields:
      #fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "host.name", "ecs.version", "input.type"]
      fields:
        - "agent.ephemeral_id"
        - "agent.hostname"
        - "agent.id"
        - "agent.type"
        - "agent.version"
        - "host.name"
        - "ecs.version"
        - "input.type"
      ignore_missing: true

filebeat.autodiscover:
  providers:
    - type: docker
      # this input section is for module eg: mongo/nginx specific
      templates:
          - condition:
            contains:
              docker.container.name: echo
          config:
            - module: nginx
              enabled: true
              access:
                input:
                  type: container
                  stream: "stdout"
                  containers.ids:
                    - ${data.docker.container.id}
              error:
                input:
                  type: container
                  stream: "stdout"
                  containers.ids:
                    - ${data.docker.container.id}
        - condition:
            contains:
              docker.container.name: darwin
          config:
            - module: mongodb
              enabled: true
              log:
                input:
                  type: docker
                  containers.ids:
                    - ${data.docker.container.id}

filebeat.inputs:
 - type: log
   # this input section is for system log files
   enabled: true
   paths:
    - /var/log/*.log
    - /var/log/messages

 - type: container
   # this input section is for general application docker/containers logs
   enabled: true
   stream: all
   paths:
     - /var/lib/docker/containers/*/*.log
   include_lines: ['"pod":']

   json.keys_under_root: true
   json.ignore_decoding_error: true
   json.add_error_key: true
   json.overwrite_keys: true
   json.message_key: log

  # Avoid parsing exceptions due to name conflicts
   processors:
  - rename:
       fields:
         - from: "service"
           to: "service-name"
         - from: "error"
           to: "service-error"
         - from: "url"
           to: "service-url"
       ignore_missing: true
       fail_on_error: false
4

Hi @john_eapen i am sorry but i meant to comment on another post. But i think the sitting you are looking for is drop_fields processor, with this setting your can drop other fields as-well except "type" and "@timestamp" since these are default fields to filebeat.
I hope this answers your question