Persisting latest state in separated index

Elastic Stack Elasticsearch
1

I have stream of data like this:

userid: 1, taskid: 1, start_time: 01:00:00, completed: 10
userid: 1, taskid: 1, start_time: 02:00:00, completed: 30
userid: 1, taskid: 1, start_time: 03:00:00, completed: 70
userid: 1, taskid: 1, start_time: 04:00:00, completed: 90

I want to save in another index just current states, so basically I need to delete previous state and create new stated based on userid and taskid. I mean only "completed" field should be updated, because start time is when the first event occurred anyway.

something like:
userid: 1, taskid: 1, start_time: 01:00:00, completed: 10
is current state, after that,
userid: 1, taskid: 1, start_time: 01:00:00, completed: 30
etc.., last event is
userid: 1, taskid: 1, start_time: 01:00:00, completed: 30, end_time: 07:00:00

Hope I made myself clear. Is this something that should be done with some scripting, or in Logstash/Filebeat while inserting data, or should I keep just one index and adjust queries to grab only current state of the field?

Thanks

2

Look at using a transform for this - Transforming data | Elasticsearch Guide [7.15] | Elastic

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.