[BUG] Metricbeat stops generating the right timestamp after a while

[vhenriquez (Victor Henriquez)]

Metricbeat 8.6 (and possibly other 8.x versions) stops sending the right timestamp after a while.

When metricbeat starts running, the timestamps sent to Logstash are correct, but after several hours without restarting the service, it starts generating wrong timestamps.

Here are a few messages received by logstash, check the debug timestamp (correct) against the one sent by metricbeat (@timestamp)

2023-03-15 05:09:57,973 DEBUG recv msg: {"@timestamp":"2023-03-14T17:42:24.462Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"8.6.2"},"agent":{"type":"metricbeat","version":"8.6
2023-03-15 05:09:57,974 DEBUG recv msg: {"@timestamp":"2023-03-14T17:42:24.462Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"8.6.2"},"event":{"module":"system","duration":45809

2023-03-15 05:11:00,384 DEBUG recv msg: {"@timestamp":"2023-03-14T17:42:24.458Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"8.6.2"},"agent":{"type":"metricbeat","version":"8.6
2023-03-15 05:11:00,385 DEBUG recv msg: {"@timestamp":"2023-03-14T17:42:24.461Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"8.6.2"},"service":{"type":"system"},"system":{"load

2023-03-15 05:13:05,014 DEBUG recv msg: {"@timestamp":"2023-03-14T17:42:24.458Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"8.6.2"},"metricset":{"name":"uptime","period":90000
2023-03-15 05:13:05,015 DEBUG recv msg: {"@timestamp":"2023-03-14T17:42:24.461Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"8.6.2"},"service":{"type":"system"},"system":{"load

[stephenb (Stephen Brown)]

Hi @vhenriquez Welcome to the community.

Interesting can you supply your metricbeat.yml and your logstash pipeline please?
What OS?
How did you install?
How long after running are you seeing this?
Are you using any persistent queues in Metricbeat or Lostash.

This is very unusual ... we will see if we can reproduce it... there are 1000s of 8.x metricbeats out there and I don't think we have seen this reported elsewhere....yet.