Substracting event fields in partitioned data

Elastic Stack Logstash
1

Hello,

I am working with a set of devices that pushes data to my database every 5 minutes. The data contains a field with the device's id and a field with a float value.

Now, for every device I would like to substract the previous value from the current value, and add this value in a new field. I figured I would use the aggregate plugin to substract the values, but I am having trouble sorting the data on the device ids. A complicating factor is that the number of devices varies and can be quite large, so putting a bunch of aggregate filters in parallel is not feasible.

Is it possible to creat such a filter in logstash? Or is logstash not a suitable tool for this problem?

Thanks in advance!

Jorrit

2

You might be able to do that using an update script in elasticsearch. You would set the document id equal to the device id so that you can reference the previous version of the document.

3

Hi Badger,

Thank you for your reply. I have been playing around with the update functionality you suggested. Unfortunately I could not make it work, since (as far as I could see) the API does not allow the usage of values of other documents, which is needed for calculating the difference between two documents. This is the same issue I encountered using Ingest nodes and Transforms.

4

If they are different documents then that would not work. As I noted, I was assuming you would set the document id equal to the device id so that you can overwrite/update the document.

You could try fetching the existing document with an elasticsearch filter.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.