Hi all,
I've tried looking for similar use cases to this, but haven't been able to find anyone doing something along the same lines (or at least, not with Kibana involved). We're currently trying to develop an ELK stack for our syslog data to replace an aging in-house system. Our production environment is essentially 2 primary data centres (I'll call them PDCs from here) in an active/passive configuration and a few satellite data centres around the world.
Having read all the warnings and various suggestions relating to replicating elasticsearch over geographically separate locations, we've come to the conclusion that we're going to have an ELK stack per PDC and get logstash to send data to 2 clusters, 1 in each PDC. The satellites only ever send syslog messages to the active PDC. However, this also means that there will be a kibana instance in each PDC. Given that kibana can only connect to one elasticsearch cluster at once, and that it stores its own data in that cluster, we need a way of replicating the kibana data between the 2 PDCs in case of switching which one is active.
I've seen that elasticsearch has the option for tribes, and consider this to be an option. Would this work given the kibana indexes would have to be named differently? Given a tribe node would have to be in each PDC, would I need to configure a tribe per DC? I'm also considering something more crude by way of running a cron and taking a snapshot of the kibana index on whichever PDC was active, then copying across and restoring to the inactive one. This seems a bit hacky though and there's probably a much more sensible solution!
Is anyone else doing something similar, given I've seen discussions on Multi DC set ups (Async Replication and Multi DC)? Or can anyone suggest a better way of doing this that I haven't thought of?
Any thoughts on this would be greatly appreciated.
Thanks!