Filebeat | Ordering Messages

Elastic Stack Beats
1

Hi,

Would appreciate any advice on how the following problem might be overcome using the available options.

We are sending logs using Filebeat and all is working well apart from the timestamp of the log message. The application producing the logs is a little imprecise (no Ms value) around the timestamps of the logs. As such we are seeing many log events with exactly the same timestamp down the second.

Eg (the timestamp is in the LogTime field):

Event 1:
"LogTime" => "2016-08-11 14:42:54",

Event 2:
"LogTime" => "2016-08-11 14:42:54",

Event 3:
"LogTime" => "2016-08-11 14:42:54",

You get the idea.

I would like to inject a unique sequence number into the document - ideally inside Filebeat itself, so that when the documents are inspected in ES it would be possible to order the events by that sequence of numbers.

The objective is to have some way to render the sequence of events in the same order they were read out of the logfile in the first place.

Would appreciate any suggestions, thanks.

2

You could use the offset to sort on which is sent with each event.

3

Brilliant. Thanks @ruflin I was removing that field. Yes, that will do the trick nicely. Thank you.

4

This topic was automatically closed after 21 days. New replies are no longer allowed.