Problem Statement -
Filebeat streaming logs via logstash into elasticsearch (7.5)
Logstash applies the filter on epoch time & update @timestamp field during data ingestion
Logstash is incorrectly ingesting the timestamp field for some records during ingestion. In short its trimming the epoch time field which creates inconsistency.
logs snippet -
1597161900000,host1,2,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
logstash conf -
dissect {
mapping => {
"message" => "%{timestamp},%{keyvalue00},%{keyvalue01},%{keyvalue02},%{keyvalue03},%{keyvalue04},%{keyvalue05},%{keyvalue06},%{keyvalue07},%{keyvalue08},%{keyvalue09},%{keyvalue10},%{keyvalue11}"
}
remove_field => "message"
}
date {
match => [ "timestamp", "UNIX_MS" ]
target => "@timestamp"
}
Expected Behaviour -
All timestamps should be correctly parsed during data ingestion to elasticsearch
Actual Behaviour -
Logstash is trimming the timestamp field during ingestion. e.g. 1597161900000 is ingested as 61900000 or 597161900000. This creates inconsistency & a no. of records are not ingested to the targeted index as intended.
PS - This behaviour is rare but does happen once in couple of days & there is no change at the source.
Can someone please explain what could be causing this unexpected behaviour & how to mitigate it?
Thanks in advance