Hi,
I'm wondering if anyone else has had this problem.
I am in the process up upgrading filebeat on our CentOS 7 servers. They are running filebeat version 6.8 and our ELK stack is 7.10.1. We are planning to upgrade to filebeat 7.11.1 and I have read the doco and followed the recommenced procedure.
From what I have found, there is some issue with the new version of filebeat pushing syslog events into our elastic cloud, as I have been able to build an elasticsearch and kibana server of the same version as what we have in the cloud, and successfully publish events there using filebeat 7.11.1.
When I try to push to our elastic cloud like we have been doing with filebeat 6.8, I have noticed that the issue seems to be specifically with the filebeat-7.11.1 pipeline as shown.
2021-02-28T20:53:56.746+1000 WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc006fb6ea0062c0c, ext:40129156929, loc:(*time.Location)(0x67397c0)}, Meta:{"pipeline":"filebeat-7.11.1-system-syslog-pipeline"}, Fields:{"agent":{"ephemeral_id":"xxxxxxxx","hostname":"xxxxxxxxxxxx","id":"12f1469f-e042-4173-b469-8eb472f52346","name":"xxxxxxxxx","type":"filebeat","version":"7.11.1"},"ecs":{"version":"1.7.0"},"env":"dev","event":{"dataset":"system.syslog","module":"system","timezone":"+10:00"},"fileset":{"name":"syslog"},"host":{"name":"xxxxxxxxxxxxxx"},"input":{"type":"log"},"log":{"file":{"path":"/var/log/messages"},"offset":124248},"message":"Feb 28 20:53:39 xxxxxxxxxxxx test","service":{"type":"system"}}, Private:file.State{Id:"native::17380476-64768", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000375860), Source:"/var/log/messages", Offset:124301, Timestamp:time.Time{wall:0xc006fb6e9fdb2da2, ext:40126339288, loc:(*time.Location)(0x67397c0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x109347c, Device:0xfd00}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [host.hostname]."}}
I can see there's a whole bunch of configuration for this, but from what I can tell this is all automatically set up by filebeat. In any case, when it connects to my test env, it sets up all the same indices, templates and pipelines as in our cloud setup.
I am able to add and see other hosts syslog events in our elastic cloud but only when using version 6.8 of filebeat and not for 7.11.1. I have also tried using filebeat 7.10.1 but saw the same error.
Any help would be appreciated.
Thanks,