I would like to process the data I have in a regular way (let's say every 10m or more) in order to create new fields without the regular intervention of the user.
To clarify more, I give you one concrete example : with logstash, I get a boolean field every few seconds telling me if a system is active or not.
15/05/2019 15:10:23 active:false
15/05/2019 15:10:26 active:true
15/05/2019 15:10:27 active:false
15/05/2019 15:10:29 active:true
15/05/2019 15:10:32 active:true
I would like to extract a new field giving me the duration while the system was active since the beginning of the day for example.
I've thought about using logstash with the elasticsearch filter plugin but doing that every second is far from being optimal.
Please give me an insight about your approach to create a new field based on data coming from different documents.