I am sending log data to elastic search directly using filebeat 7.10.
I want to drop some fields of doc using drop_fields processors before sending to ES. Filebeat is dropping some fields placed in drop_fields conf but not all. Filebeat throwing following error:
DEBUG [processors] processing/processors.go:128 Fail to apply processor client{drop_fields={"Fields":["agent","host","log","input","ecs"],"IgnoreMissing":false}}: failed to drop field [agent]: key not found; failed to drop field [host]: key not found; failed to drop field [ecs]: key not found
my filebeat.yml-------------
filebeat.inputs:
- type: log
processors:
- drop_fields:
fields: ["agent", "host", "log", "input", "ecs" ]
setup.template.settings:
index.number_of_shards: 1
setup.template:
name: "test"
enabled: true
pattern: "test-*"
setup.ilm:
enabled: false
setup.dashboards.enabled: true
setup.dashboards.index: "test-*"
setup.kibana:
host: "192.111.1.1:5601"
output.elasticsearch:
hosts: ["192.111.1.1:9200"]
FileBeat logs*********************
DEBUG [processors] processing/processors.go:128 Fail to apply processor client{drop_fields={"Fields":["agent","host","log","input","ecs"],"IgnoreMissing":false}}: failed to drop field [agent]: key not found; failed to drop field [host]: key not found; failed to drop field [ecs]: key not found
2021-02-18T06:28:40.804Z DEBUG [processors] processing/processors.go:203 Publish event: {
"@timestamp": "2021-02-18T06:28:40.804Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.10.0"
},
"message": "qqqqqqqqqqqqqqqqqqq",
"agent": {
"hostname": "usps-backend1",
"ephemeral_id": "f9239289-81fe-4fa9-ac46-8ed81e0edcc0",
"id": "e35682f8-8fa6-47e0-b4fa-af78c2170d26",
"name": "usps-backend1",
"type": "filebeat",
"version": "7.10.0"
},
"ecs": {
"version": "1.6.0"
},
"host": {
"name": "usps-backend1",
"hostname": "usps-backend1",
"architecture": "x86_64",
"os": {
"codename": "stretch",
"platform": "debian",
"version": "9 (stretch)",
"family": "debian",
"name": "Debian GNU/Linux",
"kernel": "4.9.0-11-amd64"
},
"id": "aca0b202602c41c4aed5fe533c64ff85",
"containerized": false,
"ip": [
"172.25.1.66",
"fe80::f816:3eff:fe2a:5757"
],
"mac": [
"fa:16:3e:2a:57:57"
]
}
}
