Get exiting in ES index documents to Logstash pipeline to make calculations

Elastic Stack Logstash
1

Hello
I need to create documents in ES index based on the unique IDs of the events. And when the ID of the new event will be the same as the ID of the documents already stored in the ES index, I want NOT TO overwrite it, but to add numeric values from fields in the existing document to the numeric fields of the new event. How can I do this?
I saw that different calculations can be done using ruby filter. But how I can retrieve information about existing in index document and push this information to logstash pipeline again?

2

You might be able to use an elasticsearch filter to fetch fields from an index if you want to add fields to an existing document. Alternatively, instead of writing directly to ES, you could use logstash to generate a text file that could be added to ES using the update/bulk APIs via curl.

1 Like
3

Thanks for recommending elasticsearch filter!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.