I get a flood of errors since updating the cluster and filebeat to version 7.
My setup consists of a 2 node elasticsearch cluster and a bunch of servers running filebeat with modules (system, auditd and nginx) shipping logs directly to the es cluster.
I get these Cannot write to a field alias [host.hostname].
and Can't get text on a START_OBJECT at 1:***
from every module.
Things I already tried:
-
Running filebeat setup
-
Deleting the index
-
Updating config a bit
Apr 26 08:43:13 app-flo22 filebeat: 2019-04-26T08:43:13.729Z#011WARN#011elasticsearch/client.go:526#011Cannot index event publisher.Event{....."fileset":common.MapStr{"name":"syslog"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000ec84e0), Source:"/var/log/messages", Offset:5097222783, Timestamp:time.Time{wall:0xbf28ce3fdc8b58b0, ext:4760377953, loc:(*time.Location)(0x2576ec0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x314e, Device:0xfd01}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [host.hostname]."}} Apr 26 08:43:13 app-flo22 filebeat: 2019-04-26T08:43:13.729Z#011WARN#011elasticsearch/client.go:526#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf28cf0825bbb7bc, ext:805914543076, loc:(*time.Location)(0x2576ec0)}, Meta:common.MapStr{"pipeline":"filebeat-7.0.0-nginx-access-default"}, Fields:common.MapStr{"log":common.MapStr{"file":common.MapStr{"path":"/var/log/nginx/flo_access.log"}, "offset":1455336}, "service":common.MapStr{"type":"nginx"}, "fileset":common.MapStr{"name":"access"}, "ecs":common.MapStr{"version":"1.0.0"}, "host":common.MapStr{"name":"app-flo22.host.com"}, "message":"80.240.16.174 - - [26/Apr/2019:08:43:10 +0000] \"GET /api/public/check HTTP/1.1\" 200 0 \"-\" \"-\"", "input":common.MapStr{"type":"log"}, "event":common.MapStr{"module":"nginx", "dataset":"nginx.access"}, "agent":common.MapStr{"type":"filebeat", "ephemeral_id":"e34ee83e-a281-48a5-93e8-f882756db201", "hostname":"app-flo22.host.com", "id":"e4ba4f29-da17-4d99-9403-169b03c4c1be", "version":"7.0.0"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000288270), Source:"/var/log/nginx/flo_access.log", Offset:1455430, Timestamp:time.Time{wall:0xbf28ce3fdd0c5957, ext:4768832121, loc:(*time.Location)(0x2576ec0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x5e54a, Device:0xfd01}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [source] of type [keyword] in document with id 'xd3QWGoBxrvTgY5hF3Wj'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:317"}} Apr 26 08:43:13 app-flo22 filebeat: 2019-04-26T08:43:13.729Z#011WARN#011elasticsearch/client.go:526#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf28cf0825bd4ebd, ext:805914647384, loc:(*time.Location)(0x2576ec0)}, Meta:common.MapStr{"pipeline":"filebeat-7.0.0-nginx-access-default"}, Fields:common.MapStr{"log":common.MapStr{"offset":1455430, "file":common.MapStr{"path":"/var/log/nginx/flo_access.log"}}, "service":common.MapStr{"type":"nginx"}, "input":common.MapStr{"type":"log"}, "ecs":common.MapStr{"version":"1.0.0"}, "agent":common.MapStr{"version":"7.0.0", "type":"filebeat", "ephemeral_id":"e34ee83e-a281-48a5-93e8-f882756db201", "hostname":"app-flo22.host.com", "id":"e4ba4f29-da17-4d99-9403-169b03c4c1be"}, "message":"80.240.16.174 - - [26/Apr/2019:08:43:10 +0000] \"GET /api/public/check HTTP/1.1\" 200 0 \"-\" \"-\"", "event":common.MapStr{"module":"nginx", "dataset":"nginx.access"}, "fileset":common.MapStr{"name":"access"}, "host":common.MapStr{"name":"app-flo22.host.com"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000288270), Source:"/var/log/nginx/flo_access.log", Offset:1455524, Timestamp:time.Time{wall:0xbf28ce3fdd0c5957, ext:4768832121, loc:(*time.Location)(0x2576ec0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x5e54a, Device:0xfd01}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [source] of type [keyword] in document with id 'xt3QWGoBxrvTgY5hF3Wj'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:317"}}
(First error is shortened because of char limit)