Elasticsearch cluster with nodes on different sites - Hints / Problems?

Elastic Stack Elasticsearch
1

Hello ELK community,
we consider to have an elasticsearch cluster (e.g.3 nodes in a cluster) running in different locations / sites.
For example: 2 nodes in one city and the third in another.

My questions, since this is a complete new approach for us:
Can it be done like this?
Are there specific points / problems / configurations to consider?
Are there does / und donts?
How about latency?

Any insights and / or opinions in this matter are very welcomed - thank you!
greetings, Stefano

2

Yes, it works. Just do not create a cluster that span sites. Instead create one cluster per site, and use tribe node to aggregate/federate across them.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-tribe.html

I do this with very large number of nodes, multiple data centers. There are issues at my scale, but for the size you described, tribe node will work perfectly.

3

Thank you tinie for your feedback and I am pondering about it. You advised not to span sites and to use a tribe node. Considering that we use the elk solution only to centralize logfiles (so there is not so much criticality involved in this), do you still think it would not fit to span the nodes in two cities in the same country? Where do you see the problems? Did you have bad experiences already in such a solution?
greetings, Stefano

4

Our experience with elk clusters that span physical locations was not good. Even with very fast WAN connections, we still see issues.

If you wish to centralize logs from more than one physical locations, I'd suggest shipping them to a central location and setup ELK there only. Use logstash-forwarder, or a local logstash instance. If you are concerned about redundancies, use a persistent queue, such as Kafka, redis, etc.

Or do what we do. Setup separate ELK cluster per data center. Use tribe to aggregate.

5

This is just not a good idea. Elasticsearch clusters do not like the lag introduced between nodes this way. It causes pain. The current best practice is, as @tinle says, to use separate clusters and a tribe node to federate searches between them

6

Dear Community members,
thank you for your insights. As you describe, we really should not go this way. I will very much consider your point of views - no need to have pain in production if you can avoid it.