3 node cluster suggestion

Elastic Stack Elasticsearch
1

Hi

I am trying to setup a cluster for my production environment. I will be using three nodes.

I am planning to use one node as master node with no data .
and two nodes as data nodes. (I want to use indices as one primary and one replica)

    node-1 --> only master not data
    node-2 --> only data
    node-3 --> only data

My production environment is not very critical and can afford down time.

Please let me know if I am following the correct approach. or should I add all three nodes as master eligible nodes.

2

I would recommend making the data nodes master eligible as well to get high availability. Remember to set minimum master nodes to 2.

3

@Christian_Dahlqvist Thanks for your reply. I have one more question.

  1. While forwarding logs from beatagents should I configure them to forward logs to all three nodes in a cluster or Just forward them to one main master node.

  2. Will elastic search decide to keep which node as hot node and the other one as cold node. Or I have specify some where in the configuration of ES. (I know that there is a lifecycle policy we can define to move indices to cold and freeze)

4

Dedicated master nodes should not serve traffic, so you should send requests only to the master/data nodes.

Hot/Warm/Cold architectures require you to set and use node attributes. It is not automatic. If your nodes have the same specification and you only have 2 data nodes it may not make sense to implement a hot/warm architecture.

5

Dedicated master nodes should not serve traffic, so you should send requests only to the master/data nodes.

-- Do you mean I should forward it to only data nodes.

Hot/Warm/Cold architectures require you to set and use node attributes. It is not automatic. If your nodes have the same specification and you only have 2 data nodes it may not make sense to implement a hot/warm architecture.

-- I want to use this format because. I will be installing this on 3 VM's. I will not have enough space on vms.
I am planning to attach a mount (which has more space) to one data node and use it as a cold node.
Add life-cycle management and move indices from hot to cold after 3 months. We will not be searching for old data much.

Do you think it will be useful.

6

I you have a single hot node you should probably send all data to that one as it will be doing all indexing.

Then follow the instructions in the blog post I linked to.

7

Thank you for your suggestion.

8

@Christian_Dahlqvist One last question.

If I forward logs to one hot node which does all the indexing.

I decide to keep one primary and one replica indices. Will the hot node have both primary and replica. Or will it create a primary on hot node and create a replica on warm/cold node?.

9

If you have a single hot node no replica can be assigned as primary and replica need to go on different nodes. Given the size of your cluster I would recommend a non-hot/warm setup with replica in order to gain reliability.

10

@Christian_Dahlqvist

I do not have enough space on the servers. My vm can max get space of 400GB. If there is a possibility I can send the indices to node3 after certain period of time. Then I can go with non-hot/warm setup. Please suggest me if It can done.

Node1 --400gb
node2 --400gb
node3 -- 400gb can attach a mount which can hold 800gb of data.

11

You could have two hot data nodes, which allo you to have a replica configured. When you relocate an index to the single bold node you could simply disable the replica at that points.

12

@Christian_Dahlqvist Sorry for asking this again.

If I have two hot data nodes.

I will be forwarding logs from beats to two data nodes in the configuration. So If i set my settings as 1 primary and 1 Replica. It will create this on both nodes correct which will utilize more space.

Please guide me how I can achieve this.
I need 1 primary indices to stay on node1 and move replica to node2. Do I need to set this on lifecycle management page on kibana.

One more questions. Can I make the main master node as data node or is it not a good idea.

   node-1 --> only master not data  (Can I make this a data node as well)   -- Hot
   node-2 --> only data                                                     -- Hot
   node-3 --> only data                                                     -- Warm
13

If the node has the same specification apart from one with more disk, make all master eligible and holding data. Make two form the hot zone so you can have a replica and get a bit more resiliency. Make the last node with more disk space form the warm tier and reduce replica count to 0 when you relocate shards to the warm zone.

14

Can you please guide me how I can reduce replica to 0 when relocating the shards to warm node.

15

You can handle that through ILM actions.

16

@Christian_Dahlqvist Thanks for your help. I have one last thing to ask.

I am given 3 medium sized machines. with 7.5GB RAM. I have read the documentation which says the memory should not be less than 8 GB. I have been running Elastic stack on single node on similar(7.5 RAM) sized machines.

If I go with the same size of vms for all three nodes. Do you think I will have issues going forward. In my prod env I will be monitoring approx 500 servers logs for one year.

17

If your hosts have 7.5 GB RAM you should give 50% to heap. Whether it is enough or not depends on how you index your data and optimize storage.

18

Thank you.

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.