I am working with a set of devices that pushes data to my database every 5 minutes. The data contains a field with the device's id and a field with a float value.
Now, for every device I would like to substract the previous value from the current value, and add this value in a new field. I figured I would use the aggregate plugin to substract the values, but I am having trouble sorting the data on the device ids. A complicating factor is that the number of devices varies and can be quite large, so putting a bunch of aggregate filters in parallel is not feasible.
Is it possible to creat such a filter in logstash? Or is logstash not a suitable tool for this problem?
You might be able to do that using an update script in elasticsearch. You would set the document id equal to the device id so that you can reference the previous version of the document.
Thank you for your reply. I have been playing around with the update functionality you suggested. Unfortunately I could not make it work, since (as far as I could see) the API does not allow the usage of values of other documents, which is needed for calculating the difference between two documents. This is the same issue I encountered using Ingest nodes and Transforms.
If they are different documents then that would not work. As I noted, I was assuming you would set the document id equal to the device id so that you can overwrite/update the document.
You could try fetching the existing document with an elasticsearch filter.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.