Logstash parsing timestamp incorrectly

Problem Statement -
Filebeat streaming logs via logstash into elasticsearch (7.5)
Logstash applies the filter on epoch time & update @timestamp field during data ingestion
Logstash is incorrectly ingesting the timestamp field for some records during ingestion. In short its trimming the epoch time field which creates inconsistency.

logs snippet -

1597161900000,host1,2,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0
1597161900000,host1,1,Request,proc,Process001,0,0,0.000000,0.000000,0.000000,0.000000,0

logstash conf -

  dissect  {
    mapping  => {
    "message" => "%{timestamp},%{keyvalue00},%{keyvalue01},%{keyvalue02},%{keyvalue03},%{keyvalue04},%{keyvalue05},%{keyvalue06},%{keyvalue07},%{keyvalue08},%{keyvalue09},%{keyvalue10},%{keyvalue11}"
    }
  remove_field => "message"
  }
	  date {
    match => [ "timestamp", "UNIX_MS" ]
    target => "@timestamp"
  }

Expected Behaviour -
All timestamps should be correctly parsed during data ingestion to elasticsearch

Actual Behaviour -
Logstash is trimming the timestamp field during ingestion. e.g. 1597161900000 is ingested as 61900000 or 597161900000. This creates inconsistency & a no. of records are not ingested to the targeted index as intended.
PS - This behaviour is rare but does happen once in couple of days & there is no change at the source.

Can someone please explain what could be causing this unexpected behaviour & how to mitigate it?

Thanks in advance

I very much doubt logstash is doing that. I think it more likely that you are seeing inode re-use causing filebeat to start reading a file part way through.

Thanks for pointing in the right direction. Its sorted

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.