Inconsistent timestamp format from logstash out

[dchavan]

There seems to be some inconsistency in the way logstash @timestamp is added in the out file.

{"host":"X","version":"unknown","event":{"original":"2022-10-12T15:51:22,937 WARN  [EAF11E629F8C4DF987673EAABE186797][scheduling-8] c.a.c.d.d.CertificateChangeMessagesDao: expect update count of 1, but get 0"},"message":"2022-10-12T15:51:22,937 WARN  [EAF11E629F8C4DF987673EAABE186797][scheduling-8] c.a.c.d.d.CertificateChangeMessagesDao: expect update count of 1, but get 0","path":"/usr/logs/cpc-main-stdout.log","app":"cps","filename":"cps-main-stdout.log","ip_addr":"198.X.X.X","@timestamp":"2022-10-12T15:51:23Z"}
{"host":"X","version":"unknown","event":{"original":"2022-10-12T15:51:22,937 INFO  [EAF11E629F8C4DF987673EAABE186797][scheduling-8] c.a.c.d.d.CertificateChangeMessagesDao: upsert updated 34930"},"message":"2022-10-12T15:51:22,937 INFO  [EAF11E629F8C4DF987673EAABE186797][scheduling-8] c.a.c.d.d.CertificateChangeMessagesDao: upsert updated 34930","path":"/usr/logs/cpc-main-stdout.log","app":"cpc","filename":"cpc-main-stdout.log","ip_addr":"198.X.X.X","@timestamp":"2022-10-12T15:51:23.000015Z"}
{"host":"X","version":"unknown","event":{"original":"2022-10-12T15:51:22,937 INFO  [EAF11E629F8C4DF987673EAABE186797][scheduling-8] c.a.c.d.s.GrimReaperService: processStalledJobs: certChangeMessagesIdFinal: 34930"},"message":"2022-10-12T15:51:22,937 INFO  [EAF11E629F8C4DF987673EAABE186797][scheduling-8] c.a.c.d.s.GrimReaperService: processStalledJobs: certChangeMessagesIdFinal: 34930","path":"/usr/logs/cpc-main-stdout.log","app":"cpc","filename":"cpc-main-stdout.log","ip_addr":"198.X.X.X","@timestamp":"2022-10-12T15:51:23.000029Z"}

As seen above the @timestamp field has one value which is "2022-10-12T15:51:23Z" (discrepancy) and the rest are 2022-10-12T15:51:23.SSSZ.
What could be the reason for this? Also, can using a ruby filter fix this? I would like to get some example config to have all the timestamps in "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" format.

[Badger]

How are you setting [@timestamp]?

[dchavan]

I don't have any specific configuration for @timestamp in the logstash config file. It's probably set to default. I am using logstash V8.3

[Rios]

"@timestamp":"2022-10-12T15:51:23Z"
"@timestamp":"2022-10-12T15:51:23.000015Z"
"@timestamp":"2022-10-12T15:51:23.000029Z"

These are LS time when messages were received in nanoseconds. The value :23Z means :23.000000Z" Nanosec are since ELK 8.x If you want to change, use the date plugin

[dchavan]

I do want the LS time to stay in nanoseconds, but I want to replace the one that gets truncated back to ":23.000000Z"

Would date plugin allow me to do that?

[Rios]

Not sure is it possible. Test

date
{
  match => [ "@timestamp", "ISO8601" ]
  target => "@timestamp"
}