5 node cluster setup for elk

Elastic Stack Elasticsearch
1

Hello,
So, I have 5 nodes available of RHEL, I have to setup ELK as cluster on these nodes,

I did this before with 2 masters, 3 data nodes, with each master will have logstash and kibana. What happened was the cluster failed once in a day only on one specific master node, and because I set minimum master to 2 the logs were not written as the master used to go down.
Later,
I changed the architecture to 1 master 1 client node and 3 data nodes but the same happened again and the cluster went down.
Background about the amount of data I was receiving was once I start the cluster first time it would gather nearly 150-200 gb of indices (no of replica was 1) but as days passed by data would fluctuate and generated index of around 3-6gb per day, But as the cluster went down and I restart it, again the indices would reach to nearly 150gb, this would cause cluster to come down as searches got slow.
What do you guys think about such situation?
What should be my cluster architecture for these situation if I have around 100-120 client hitting elastic cluster?

My bad that I cannot provide the logs as the cluster is wiped.
Thank you

2

There are a few simple guidelines I would recommend you follow:

  • Always have at least 3 master eligible nodes and make sure to set minimum_master_nodes to the correct value in Elastichsearch 6.x and earlier.
  • Just because you CAN have dedicated node types does not mean you SHOULD. For small clusters it is generally fine to deploy nodes that have all roles. If your cluster gets larger and/or busy you can consider introducing dedicated master nodes.
  • If you have dedicated master nodes these should be left alone to manage the cluster and NOT serve traffic. You are using dedicated master nodes for increased stability, not performance. This is why they typically can be quite small.
  • Always make sure that all nodes in the cluster use exactly the same version (except during upgrades) and and OS type.
  • You can run other services on the nodes, e.g. Logstash and Kibana, but this may make it more difficult to troubleshoot performance issues as they will compete for the same resources. Therefore ideally avoid doing this whenever possible.