Lets say I have 3 records with same correlation id. The only thing that is different is statuses of these records (active, errornous, issued). I want the flow to be like that:
- 1000 records are generated sent to elasticsearch via filebeat
- Each record's _id=CorrelationId
- Another 1000 records are generated.
- If there are records with existing correlation id in the index I want them to be replaced (or updated if you can call so).
So in order to achieve that I have 2 possible solutions that are only assumptions:
- Keep all records without replacing and filter them with query in kibana (is it possible?)
- Replace old records with the new ones. (Maybe filebeat automatically overwrites records when it sees that _id already exists)
I would be happy if you could provide me with some guidances how can I achieve that? I have been searching for the solution for quite a long time.