Hi,
we are ingesting data using checkpoint filebeat module.
Some records are not ingested and gives this error:
May 17 09:24:45 XXXX filebeat[3110397]: 2022-05-17T09:24:45.674Z#011WARN#011[elasticsearch]#011elasticsearch/client.go:414#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.May, 17, 9, 24, 45, 212157445, [time.Local](https://time.local/)), M
eta:{"pipeline":"filebeat-7.16.2-checkpoint-firewall-pipeline","truncated":false}, Fields:{"agent":{"ephemeral_id":"bb138b82-8f1b-4f86-90ba-651651353b96","hostname":"XXXX","id":"27347e9a-697f-4a7a-87c7-5c6e1a06db6e","name":"XXXX","type":"filebeat",
"version":"7.16.2"},"ecs":{"version":"1.12.0"},"event":{"dataset":"checkpoint.firewall","module":"checkpoint","timezone":"+00:00"},"fileset":{"name":"firewall"},"input":{"type":"udp"},"log":{"source":{"address":"X.X.X.X:XXXX"}},"message":"\u003c134\u003e1 2022
-05-17T09:24:44Z MNGcheckpoint CheckPoint 9052 - [action:\"Accept\"; contextnum:\"1\"; flags:\"802832\"; ifdir:\"inbound\"; ifname:\"bond0.2\"; logid:\"6\"; loguid:\"{0xb0d57ecf,0x37bfe3a5,0x106d0d90,0xb67980}\"; origin:\"X.X.X.X\"; originsicname:\"CN=XXXX,O=
Fwptv..h63sbj\"; sequencenum:\"204\"; time:\"1652779484\"; version:\"5\"; __nsons:\"0\"; __p_dport:\"0\"; __policy_id_tag:\"product=VPN-1 \u0026 FireWall-1[db_tag={531946FA-D4CA-EF4B-BD26-AEC6A1252661};mgmt=fwmng;date=1652697487;policy_name=XXXXX\\]\"; __pos:\"7\"; bytes:\"9445\"; client_inbound_bytes:\"313\"; client_inbound_interface:\"bond0.2\"; client_inbound_packets:\"3\"; client_outbound_bytes:\"9132\"; client_outbound_packets:\"18\"; context_num:\"1\"; elapsed:\"0\"; hll_key:\"150280794255197796
77\"; packets:\"21\"; product:\"Log Update\"; segment_time:\"1652779479\"; server_inbound_bytes:\"9235\"; server_inbound_packets:\"9\"; server_outbound_bytes:\"353\"; server_outbound_interface:\"bond1.863\"; server_outbound_packets:\"7\"; start_time:\"1652779479\"
]\n","service":{"type":"checkpoint"},"tags":["checkpoint-firewall","forwarded"]}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse"
,"caused_by":{"type":"illegal_argument_exception","reason":"data stream timestamp field [@timestamp] is missing"}}, dropping event!
Filebeat & Elastic version: 7.16.2
The error appears in filebeat log
How can we workaround this?
Is it possible to add an arbitrary timestamp where missing before indexing?
Thanks in advance!