Filebeat-7.10.2-mongodb-log-pipeline: cannot write to a field alias [cloud.availability_zone]

Elastic Stack Beats
1

Dear experts,
after migration from 6.8.13 to 7.10.2, MongoDB events logged are no more properly parsed by the factory mongodb ingest pipelines.
To be honest, I have no idea about the root cause. Many thanks for your help.

Here is an extract from Filebeat log files on MongoDB cluster node:

2021-03-10T09:30:38.185+0100    WARN    [elasticsearch] elasticsearch/client.go:408     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc00a3ecb065d4eea, ext:73104681731, loc:(*time.Location)(0x64d0ce0)}, Meta:{"pipeline":"filebeat-7.10.2-mongodb-log-pipeline"}, Fields:{"agent":{"ephemeral_id":"488341fb-a559-4f84-b127-28a2f2620734","hostname":"i-002c4ce9-hidden-database-server-15862443171.novalocal","id":"14bb78ce-e144-4aa5-9651-72441f09fe4f","name":"192.168.2.31","type":"filebeat","version":"7.10.2"},"cloud":{"availability_zone":"eu-west-0a","instance":{"id":"i-002c4ce9","name":"hidden-database-server-15862443171.novalocal"},"machine":{"type":"s3.xlarge.4"},"provider":"openstack"},"ecs":{"version":"1.5.0"},"event":{"dataset":"mongodb.log","module":"mongodb"},"fileset":{"name":"log"},"host":{"architecture":"x86_64","containerized":false,"hostname":"i-002c4ce9-hidden-database-server-15862443171.novalocal","id":"0a23dfe41753481fb353c17b22951382","ip":["192.168.2.31"],"mac":["fa:16:3e:e9:8f:ed"],"name":"192.168.2.31","os":{"codename":"Core","family":"redhat","kernel":"3.10.0-1160.15.2.el7.x86_64","name":"CentOS Linux","platform":"centos","version":"7 (Core)"}},"input":{"type":"log"},"log":{"file":{"path":"/mondata/log/mongod.log-20210310"},"offset":7912042},"message":"2021-03-09T15:52:56.676+0100 I NETWORK  [conn17040] end connection 127.0.0.1:57984 (4 connections now open)","service":{"type":"mongodb"}}, Private:file.State{Id:"native::136017621-64784", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000be0b60), Source:"/mondata/log/mongod.log-20210310", Offset:7912150, Timestamp:time.Time{wall:0xc00a3eca52028631, ext:70300058716, loc:(*time.Location)(0x64d0ce0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x81b76d5, Device:0xfd10}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [cloud.availability_zone]."}}
2

I also would like to add that this is a global error ... whichever the log file filebeat is working on.
Here is an extra example with a postfix logfile when ingest pipeline are not involved at all:

2021-03-10T10:54:22.799+0100    WARN    [elasticsearch] elasticsearch/client.go:408     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc00a43b36f464a44, ext:15377385273, loc:(*time.Location)(0x64d0ce0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"ba3a2e1e-cc9d-4a41-9c86-b0c19c61f171","hostname":"bastion.novalocal","id":"659f54e7-e435-4dd9-9218-98c9b990b191","name":"192.168.2.5","type":"filebeat","version":"7.10.2"},"cloud":{"availability_zone":"eu-west-0a","instance":{"id":"i-00127611","name":"bastion.novalocal"},"machine":{"type":"s3.large.2"},"provider":"openstack"},"ecs":{"version":"1.6.0"},"host":{"architecture":"x86_64","containerized":false,"hostname":"bastion.novalocal","id":"c9fbf7d16275444780e11704daee8f08","ip":["192.168.2.5","10.228.202.66"],"mac":["fa:16:3e:98:03:bc","fa:16:3e:e1:32:59"],"name":"192.168.2.5","os":{"codename":"Core","family":"redhat","kernel":"3.10.0-957.1.3.el7.x86_64","name":"CentOS Linux","platform":"centos","version":"7 (Core)"}},"input":{"type":"log"},"log":{"file":{"path":"/var/log/maillog"},"offset":2761918},"message":"Mar 10 10:54:18 bastion postfix/smtp[4966]: 35BBE392: to=\u003cpatrick.masson@orange.com\u003e, relay=relais-nor19.orange.com[80.12.70.19]:25, delay=0.17, delays=0.05/0/0.03/0.09, dsn=5.7.1, status=bounced (host relais-nor19.orange.com[80.12.70.19] said: 554 5.7.1 \u003cPFS-KLIF-FLEX.datasync@orange.com\u003e: Sender address rejected: domain use is reserved (in reply to RCPT TO command))"}, Private:file.State{Id:"native::916-64773", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000830340), Source:"/var/log/maillog", Offset:2762289, Timestamp:time.Time{wall:0xc00a43af9d7c889c, ext:78950336, loc:(*time.Location)(0x64d0ce0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x394, Device:0xfd05}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [cloud.availability_zone]."}}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.