Thnx for your reply.
The cluster is still in designphase, so maybe that's the easy part
I don't think the replication is needed. between the clusters. I think it is enough when each location records it's own data. You see, we use ES to monitor a cluster of servers which act as some kind of requestgateway to our platform. At this moment I have 8 of those servers on each location and this will be doubled during the course of this year. These 32 servers will act as a gateway for up to 3 mln clients in the busiest hours.
You can imagine this number of clients generate way to much logging for engineers to check logs per server to pinpoint issues. Hence the use of ES/kibana. By the way, yes, the locations are 2 different datacenters in 2 different regions. I recall I read on different locations in the web that spanning a cluster (or replicating nodes) over wan is not recommended, and the more data is involved the less recommended it is?
Kibana is brilliant in it's simplest form by just being able to quickly search through these loglines in a certain timeperiod. Again, brilliant for troubleshooting.
Back to the usecase. When 1 location is somehow totally down (for example poweroutage, servers will keep running, but communication may be interrupted) all customertraffic will pass through the other location. In that case that other location still needs to be monitored. When both locations are active, I want the Kibana interface to run my search query over the logs of that total of 32 machines. When 1 location is down, it's just missing that 1 location. A problem for our services, but should not be a problem for kibana then
So the data may be seperate, but for the view it shuld look like 1 index. I read something about indexaliases as well, but did not have the time to drill deaper on that yet.
Not need for comined data like in joins, but when both locations are up I'ld like to see both locations combined in the listing on the discovery page and in our graphs.
As said, I will need to have 2 logstashes and 2 kibana's as well, becuase these also need to stay functioning when the other location s down.
When both locations are up, both kibana's should give the same results. When 1 of the sides is down, the kibana of the other side should just show the logging of it's own side.
Hope this clears things a bit. This is the idea I have now. May change during the way, but I have to look at performance as well, so replication across locations is not what I have in mind at the first place.