5 node cluster architecture recommendations

Elastic Stack Elasticsearch
#1

I know I have seen similar questions and am still struggling to figure out the best way to architect a 5 node cluster. This is for geo-spatial data and we expect very heavy indexing, roughly 200gb of data in 9 indexes, queries around 1100/second and, of course, minimal latency and ability to expand as needed. I want to follow best practices but it seems doing so would require a lot of overhead and resources (3 masters, however many data nodes necessary, a separate cluster just for marvel indices, Kibana on it's own etc.) and we are really trying to keep to the 5 nodes to start. Is it better to have 3 dedicated masters and two data nodes maxed out at 64GB of Ram with a 31GB heap? Or have one dedicated master and 4 data/master nodes? What about sending marvel indices elsewhere? Would you anticipate an issue if the marvel indices were actually sent to a cluster at another site (so all marvel indices could be consolidated across my environments)?

Any assistance with suggestions/explanations would be very helpful. Thank you.

#2

You can try the following with 5 nodes

  • 3 master nodes + data nodes (cluster A)
  • 1 client node + kibana (cluster A)
  • 1 marvel + master node + data node (cluster B)

Cluster A

  • you definitely need 3 master nodes

  • with 3 data nodes to start with, you can do 3 shards + 1 replicas per index
    (you need to index your data to see if 3 shards + 1 replica per index is good or not)

  • translate 200GB of data to X documents

    • estimate how many documents per index
    • estimate how many documents per shards (each shard can hold upto 2B documents)
      • if you need to expand cluster A, add more data node(s)

  • client node helps reducing the load for the master nodes and connect kibana to this client node for visualization

Cluster B

  • Starting from v2.x (I think) Marvel can be deployed in its own cluster and is capable of monitor other cluster(s). By doing this way, Marvel can create its daily indices in cluster B and all traffic for viewing data is in cluster B. There is still traffic from cluster A to B because it needs to, can't change that at all.

  • If you need to expand cluster B down the road, just add more data node and try to get to a least 3 master nodes if you could.

  • Starting with one node is okay for what you are doing with an acceptable risk that is this node goes down, you'll loose something or down for a period of time until it's up. If you absolutely have to have it running all the time and can't afford having it down for a period of time or losing data, you need more HW.

1 Like
#3

I wouldn't split them in this instance, at least not yet.

That's a pretty high query rate but not a really massive dataset. I'd look at making all 5 master+data and then spreading the load accordingly as you get the max out of the servers. Don't use one master only node, never ever do that.

If you can have another smaller marvel node then do that, otherwise be super aggressive with removing old data so that it doesn't take too much resources. Note that you cannot have a single marvel cluster for multiple end-point clusters unless you have a commercial license, the free version doesn't support this.

Also make sure you use 2.X as there are heaps of geo improvements in there that will help.

1 Like
#4

I agree about not splitting them now but I think the data set in this case could grow fast, it's better to split now so it can be easily expanded (and yes, it takes away one server from a 5-node cluster)

Regarding Marvel, it's good to know that. I think I ran into that problem when I was trying to setup Marvel but did not know why.

#5

I guess that's something the OP can comment on :slight_smile:

#6

Thank you for the tip about the marvel cluster. I did not see that limitation in the documentation and do not want to have to rely on always having a commercial license. We are on the latest version.

As far as the single master I only meant have 1 node that was only a dedicated master and a couple other nodes that were both master and data. Is there any issue with that? I would assume you could rely on the standalone master electing itself master most of the time if you configured it correctly and were careful about the way you bring the environment up.

#7

I have not set up my marvel cluster separately yet so have nothing to add. I am still messing around with the configs though so may do so and will report back on issues if I put it at a different site. Thanks all for your input!

#8