Hello all,
I'm trying to ingest data from a "data" directory into Elasticsearch using Filebeat. When I manually upload these data files through Filebeat I have no problems.
If I use that same mapping for my .conf file and try reading in the file I get this error. I've searched through the forum and found an issue very similar, but their solution didn't work for my problem.
I have the error below. Even though it says that it can not index the event, Filebeat was able to with a manual upload through the gui before. I thought maybe this is a false positive error flag.
Any thoughts would be greatly appreciated.
Thank you
Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfb703e4f430f60d, ext:38571866, loc:(*time.Location)(0x5b230e0)}
2020-06-30T15:52:52.894-0400 WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfb703e4f430f60d, ext:38571866, loc:(*time.Location)(0x5b230e0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"e95130a7-ca0e-4de4-9130-f69b61cac698","hostname":"localhost","id":"bsb875c1-6a4b-4bb1-bb8a-f4da88f0245c","name":"localhost","type":"filebeat","version":"7.8.0"},"ecs":{"version":"1.5.0"},"host":{"name":"localhost"},"input":{"type":"log"},"log":{"file":{"path":"/data/ingest/test/data6-test4.json"},"offset":0},"message":"{\"flow_version\":12,\"ip_version\":4,\"tuple\":{\"src_ip\":\"10.104.8.16\",\"dst_ip\":\"10.104.11.255\",\"protocol\":17,\"src_port\":138,\"dst_port\":138},\"continued\":false,\"continuation\":false,\"direction\":\"Internal\",\"bidirectional\":false,\"vlan\":1010,\"scan\":false,\"start_time_usec\":292632,\"end_time_sec\":1591123480,\"end_time_usec\":292632,\"packets\":1,\"bytes\":247,\"payload_bytes\":201,\"src_role\":\"Unknown\",\"metadata\":{\"src_host_name\":[\"taxmd02-veeam-v\"],\"src_host_names\":[{\"domain_name\":\"taxmd02-veeam-v\",\"active_provenance\":[900,1591122621],\"passive_provenance\":null}]}}"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000684340), Source:"/data/ingest/test/data6-test4.json", Offset:550, Timestamp:time.Time{wall:0xbfb703e4f42dae06, ext:38356819, loc:(*time.Location)(0x5b230e0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x2600b0, Device:0x802}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"field [end_time_sec] not present as part of path [end_time_sec]"}