How to compare the current event with the previous 2 events in logstash?

Elastic Stack Logstash
1

I am receiving logs every 5 minutes from many devices.
In the log, there is field called crc_error which holds numeric value

For a particular Device, if the value of the crc_error field increases consecutively for 3 logs (lets say at 12:00, 12:05 and 12:10) then a field with a warning message should be added to the log received at 12:10.

I want to compare the value of the field crc_error with the value of crc_error of the previous log. If the previous value is smaller than the current value then I want to compare the previous value with its previous value.

Please help me to achieve this using logstash.

Thanks.

2

I would do that in a ruby filter. Use a pair of hashes, with the device id as the key, to store the previous two values. Do the comparisons, use event.set to add field to the event if needed, then shift the last but one value to the last but two hash and overwrite the last but one value with the current value.

I would recommend that you run with pipeline.workers to 1 so that the same thread processes all of the events.

logstash may not preserve event order, but if they are 5 minutes apart I do not think you will have issues with that.

3

Hi@Badger,

Thanks you for the reply.

I am thinking of using ElasticSearch filter plugin in logstash to fetch the previous 2 events from the index.
Will I be able to pass the fields (DeviceName and crc_error) dynamically in the elasticsearch query to fetch the previous events?

Can I use the query, which I built using kibana dev tools, in elasticsearch filter? Or there will be a change in format?

Is it possible? Please suggest

4

That is another possibility.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.