Stream live not updating

Elastic Observability Logs
1

Hi

When using the Stream Live feature the logs do not get updated in realtime, the logs are there because a page refresh displays them.
I think the key is that each time I load up the Logs app the newest/latest entry is:
2019-12-31 23:59:53.000 INFO: xxxxx daemon running

Everything behind that is up to date i.e.
2019-01-10 12:29:21.587 Jan 10 12:29:20 localhost

So I guess something is stuck somewhere - is there a file that checkpoints and needs clearing or similar?

Regards

Ed

2

Hi @aviator,

can you check whether the timestamp in your original log messages contain a year?

3

one does and one does not, perhaps the latter is the one being filtered on?

"@timestamp": "2019-01-10T15:11:36.000Z",
"system": {
"syslog": {
"hostname": "xxxxx",
"pid": "13793",
"program": "haproxy(HAProxy)",
"message": "INFO: haproxy daemon running",
"timestamp": "Jan 10 15:11:36"

4

Yes, timestamp is what is read from the text and @timestamp is how Elasticsearch has interpreted it at indexing time. So the problem is that the original timestamp is ambiguous and its interpretation depends on the time of indexing. This becomes obvious when crossing into a new year.

My current recommendation would be to fix the incorrect timestamps using a update-by-query operation. Adapting the log producer's timestamp format to include a year would prevent that from happening again at the beginning of the next year.

We're currently discussing other ways in which we can make such situations less problematic.

5

Ahh many thanks, that all makes perfect sense now! Will have a look the the update query and see what can be done. Going forward this is something I will out for when adding log streams,

again, my thanks to you for concise and accurate assistance

Regards

Ed

6

We have created beats issue #9995 to track this scenario.

Let us know if the update query doesn't work as hoped.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.