3 node cluster suggestion

(fayaz) #1

Hi

I am trying to setup a cluster for my production environment. I will be using three nodes.

I am planning to use one node as master node with no data .
and two nodes as data nodes. (I want to use indices as one primary and one replica)

    node-1 --> only master not data
    node-2 --> only data
    node-3 --> only data

My production environment is not very critical and can afford down time.

Please let me know if I am following the correct approach. or should I add all three nodes as master eligible nodes.

(Christian Dahlqvist) #2

I would recommend making the data nodes master eligible as well to get high availability. Remember to set minimum master nodes to 2.

(fayaz) #3

@Christian_Dahlqvist Thanks for your reply. I have one more question.

  1. While forwarding logs from beatagents should I configure them to forward logs to all three nodes in a cluster or Just forward them to one main master node.

  2. Will elastic search decide to keep which node as hot node and the other one as cold node. Or I have specify some where in the configuration of ES. (I know that there is a lifecycle policy we can define to move indices to cold and freeze)

(Christian Dahlqvist) #4

Dedicated master nodes should not serve traffic, so you should send requests only to the master/data nodes.

Hot/Warm/Cold architectures require you to set and use node attributes. It is not automatic. If your nodes have the same specification and you only have 2 data nodes it may not make sense to implement a hot/warm architecture.

(fayaz) #5

Dedicated master nodes should not serve traffic, so you should send requests only to the master/data nodes.

-- Do you mean I should forward it to only data nodes.

Hot/Warm/Cold architectures require you to set and use node attributes. It is not automatic. If your nodes have the same specification and you only have 2 data nodes it may not make sense to implement a hot/warm architecture.

-- I want to use this format because. I will be installing this on 3 VM's. I will not have enough space on vms.
I am planning to attach a mount (which has more space) to one data node and use it as a cold node.
Add life-cycle management and move indices from hot to cold after 3 months. We will not be searching for old data much.

Do you think it will be useful.

(Christian Dahlqvist) #6

I you have a single hot node you should probably send all data to that one as it will be doing all indexing.

Then follow the instructions in the blog post I linked to.

(fayaz) #7

Thank you for your suggestion.

(fayaz) #8

@Christian_Dahlqvist One last question.

If I forward logs to one hot node which does all the indexing.

I decide to keep one primary and one replica indices. Will the hot node have both primary and replica. Or will it create a primary on hot node and create a replica on warm/cold node?.

(Christian Dahlqvist) #9

If you have a single hot node no replica can be assigned as primary and replica need to go on different nodes. Given the size of your cluster I would recommend a non-hot/warm setup with replica in order to gain reliability.

(fayaz) #10

@Christian_Dahlqvist

I do not have enough space on the servers. My vm can max get space of 400GB. If there is a possibility I can send the indices to node3 after certain period of time. Then I can go with non-hot/warm setup. Please suggest me if It can done.

Node1 --400gb
node2 --400gb
node3 -- 400gb can attach a mount which can hold 800gb of data.

(Christian Dahlqvist) #11

You could have two hot data nodes, which allo you to have a replica configured. When you relocate an index to the single bold node you could simply disable the replica at that points.

(fayaz) #12

@Christian_Dahlqvist Sorry for asking this again.

If I have two hot data nodes.

I will be forwarding logs from beats to two data nodes in the configuration. So If i set my settings as 1 primary and 1 Replica. It will create this on both nodes correct which will utilize more space.

Please guide me how I can achieve this.
I need 1 primary indices to stay on node1 and move replica to node2. Do I need to set this on lifecycle management page on kibana.

One more questions. Can I make the main master node as data node or is it not a good idea.

   node-1 --> only master not data  (Can I make this a data node as well)   -- Hot
   node-2 --> only data                                                     -- Hot
   node-3 --> only data                                                     -- Warm
(Christian Dahlqvist) #13

If the node has the same specification apart from one with more disk, make all master eligible and holding data. Make two form the hot zone so you can have a replica and get a bit more resiliency. Make the last node with more disk space form the warm tier and reduce replica count to 0 when you relocate shards to the warm zone.

(fayaz) #14

Can you please guide me how I can reduce replica to 0 when relocating the shards to warm node.

(Christian Dahlqvist) #15

You can handle that through ILM actions.

(fayaz) #16

@Christian_Dahlqvist Thanks for your help. I have one last thing to ask.

I am given 3 medium sized machines. with 7.5GB RAM. I have read the documentation which says the memory should not be less than 8 GB. I have been running Elastic stack on single node on similar(7.5 RAM) sized machines.

If I go with the same size of vms for all three nodes. Do you think I will have issues going forward. In my prod env I will be monitoring approx 500 servers logs for one year.

(Christian Dahlqvist) #17

If your hosts have 7.5 GB RAM you should give 50% to heap. Whether it is enough or not depends on how you index your data and optimize storage.

(fayaz) #18

Thank you.