Unable to Drop desired fields using drop_fields processors in filebeat

Elastic Stack Beats
1 
I am sending log data to elastic search directly using filebeat 7.10.
I want to drop some fields of doc using  drop_fields processors before sending to ES. Filebeat is dropping some fields placed in drop_fields conf but not all. Filebeat throwing following error:

DEBUG	[processors]	processing/processors.go:128	Fail to apply processor client{drop_fields={"Fields":["agent","host","log","input","ecs"],"IgnoreMissing":false}}: failed to drop field [agent]: key not found; failed to drop field [host]: key not found; failed to drop field [ecs]: key not found

my filebeat.yml-------------
filebeat.inputs:
- type: log
  processors:
    - drop_fields:
        fields: ["agent", "host", "log", "input", "ecs" ]

setup.template.settings:
  index.number_of_shards: 1

setup.template:
  name: "test"
  enabled: true
  pattern: "test-*"

setup.ilm:
  enabled: false

setup.dashboards.enabled: true
setup.dashboards.index: "test-*"

setup.kibana:
host: "192.111.1.1:5601"

output.elasticsearch:
  hosts: ["192.111.1.1:9200"]


FileBeat logs*********************
DEBUG	[processors]	processing/processors.go:128	Fail to apply processor client{drop_fields={"Fields":["agent","host","log","input","ecs"],"IgnoreMissing":false}}: failed to drop field [agent]: key not found; failed to drop field [host]: key not found; failed to drop field [ecs]: key not found
2021-02-18T06:28:40.804Z	DEBUG	[processors]	processing/processors.go:203	Publish event: {
  "@timestamp": "2021-02-18T06:28:40.804Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.0"
  },
  "message": "qqqqqqqqqqqqqqqqqqq",
  "agent": {
    "hostname": "usps-backend1",
    "ephemeral_id": "f9239289-81fe-4fa9-ac46-8ed81e0edcc0",
    "id": "e35682f8-8fa6-47e0-b4fa-af78c2170d26",
    "name": "usps-backend1",
    "type": "filebeat",
    "version": "7.10.0"
  },
  "ecs": {
    "version": "1.6.0"
  },
  "host": {
    "name": "usps-backend1",
    "hostname": "usps-backend1",
    "architecture": "x86_64",
    "os": {
      "codename": "stretch",
      "platform": "debian",
      "version": "9 (stretch)",
      "family": "debian",
      "name": "Debian GNU/Linux",
      "kernel": "4.9.0-11-amd64"
    },
    "id": "aca0b202602c41c4aed5fe533c64ff85",
    "containerized": false,
    "ip": [
      "172.25.1.66",
      "fe80::f816:3eff:fe2a:5757"
    ],
    "mac": [
      "fa:16:3e:2a:57:57"
    ]
  }
}
2 
Problem is solved now.
I was configuring processor at rwong location ki filebeat.yml.
```
3

Can you post the config changes?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.