While sending data from a logfile using file beat through ingest pipeline to index in Elasticsearch,
some additional fields not present in the concerned log file is also getting populated.
These data are mostly related to the host machine from which the log is send.
It is shown as given below
"ecs": {
"version": "1.11.0"
},
"host": {
"containerized": false,
"ip": [
"192.168.1.5",
"fe80::cd8f:a19c:a6b2:2628"
],
"mac": [
"c0:25:a5:6f:66:85",
"80:b6:55:95:69:c1"
],
"hostname": "shi-Latitude-3510",
"architecture": "x86_64",
"os": {
"platform": "ubuntu",
"version": "20.04.3 LTS (Focal Fossa)",
"family": "debian",
"name": "Ubuntu",
"kernel": "5.11.0-40-generic",
"codename": "focal",
"type": "linux"
},
"id": "3104710ca45f478eadec013d783b2b1c",
"name": "shi-Latitude-3510"
},
"agent": {
"id": "058ff2af-04fe-470d-bbe9-9a87caad3719",
"name": "shi-Latitude-3510",
"type": "filebeat",
"version": "7.15.0",
"hostname": "shi-Latitude-3510",
"ephemeral_id": "d56a8699-657a-4c34-96d3-0b379ced4e7b"
},
"log": {
"offset": 144730231,
"file": {
"path": "/home/shi/logfortifull/10.101.200.1--fortigate--traf-forw.log"
}
}
}
I need to remove them all.
For testing , I tried to remove one of them (ecs.version),using the filebeat configuration as follows
paths:
#- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
- /home/shi/logfortifull/10.101.200.1--fortigate--utm-virs.log
jason.keys_under_root: true
jason.message_key: log
encoding: utf-8
fields:
type: "otherfgdrp-virs"
processors:
- drop_fields:
fields: ["ecs.version"]
But it gives the following error , and is not removing the field. Field ecs.version is available in the index and is searchable from kibana also.
2021-11-30T05:05:36.395+0530 DEBUG [processors] processing/processors.go:128 Fail to apply processor client{drop_fields={"Fields":["ecs.version"],"IgnoreMissing":false}}: failed to drop field [ecs.version]: key not found
2021-11-30T05:05:36.395+0530 DEBUG [processors] processing/processors.go:203 Publish event: {
How to remove the fields ? Is there any way of removing them from filebeat stage itself using
host.* ,
agent.*
log.*
etc
thanks and rgards
shini