The use case is that we want to setup Active Active architecture in ELK using two clusters in two different regions. We will index the event to both the local elasticsearch cluster and remote elasticsearch cluster.
To achieve failure-recovery, I wanted to use DLQ (Dead Letter Queue) feature in case Elasticsearch is down so I can store them somewhere else in the local cluster until elasticsearch is back up and reprocess those events and re-index them.
After I read through the docs, it seems this is not possible because elasticsearch has to respond with either 400 or 404 to send the event to the DLQ. Is there any other option to achieve this kind of setup with failure-recovery?
What I thought of is to replace my output stages with custom logic in filter stage to index the event and in case elasticsearch is down, I can index the event somewhere else, but I don't know if any problem would appear from that or any considerations I need to have to achieve a performant indexing as I would using output stage.
Any ideas are appreciated.