Hi All
I have a large index that is populated using logstash with a Kafka input ~200m documents in it. We are building a pretty conmplex dashboard based on the data from that index
I need some guidance on how I can build a transform to achieve the following :
- contain only open or closed records, the source index can have records moving from open to closed or out of scope and back again.
- if the record moves into or out of open or closed disposition the destination index should reflect that change. ie add new document if moved from out of scope to open and remove it moved from closed to out of scope.
- be close to real time as the source index is 5m or 10m difference is acceptable.
The index does have a unique id for record , a @timestamp from logstash and a time field called ingest_timestamp from the ingest_pipeline.
I have a transform running at the moment but it seems to not be able to handle the movement between different dispositions when the filter for the transform is only capturing disposition of open or closed
Regards
Tommy