I have stream of data like this:
userid: 1, taskid: 1, start_time: 01:00:00, completed: 10
userid: 1, taskid: 1, start_time: 02:00:00, completed: 30
userid: 1, taskid: 1, start_time: 03:00:00, completed: 70
userid: 1, taskid: 1, start_time: 04:00:00, completed: 90
I want to save in another index just current states, so basically I need to delete previous state and create new stated based on userid and taskid. I mean only "completed" field should be updated, because start time is when the first event occurred anyway.
something like:
userid: 1, taskid: 1, start_time: 01:00:00, completed: 10
is current state, after that,
userid: 1, taskid: 1, start_time: 01:00:00, completed: 30
etc.., last event is
userid: 1, taskid: 1, start_time: 01:00:00, completed: 30, end_time: 07:00:00
Hope I made myself clear. Is this something that should be done with some scripting, or in Logstash/Filebeat while inserting data, or should I keep just one index and adjust queries to grab only current state of the field?
Thanks