Help with create true ES cluster

Elastic Stack Elasticsearch
1

Hello!
I have the next equipment:

virtual machine 1:
12 CPU E5-2698 v4 @ 2.20GHz with 16Gb memory
This server contains:
logstash with -Xms256m -Xmx1g
elastic (node.master: true node.data: false node.ingest: true) with -Xms4g -Xmx4g
kibana

hardware machine 1:
24 CPU X5660 @ 2.80GHz with 96Gb memory and 18Tb filesystem
This server contains:
elastic (node.master: false node.data: true node.ingest: true) with -Xms30g -Xmx30g

hardware machine 2:
16 CPU E5620 @ 2.40GHz with 96Gb memory and 18Tb filesystem
This server contains:
elastic (node.master: false node.data: true node.ingest: true) with -Xms30g -Xmx30g

Elastic, logstash and kibana version 6.1.2 with x-pack (basic license)

My index rate 2000-3000 events per second with burst up to 15000-25000 events per second.

When I have burst , My master elastic is down with error "out of memory".

Than I added new elastic node:

virtual machine 2 :
4 CPU E5-2698 v4 @ 2.20GHz with 8Gb memory
This server contains:
elastic (node.master: false node.data: false node.ingest: true) with -Xms6g -Xmx6g

and change elastic on virtual machine 1 to elastic (node.master: true node.data: false node.ingest: true) with -Xms2g -Xmx2g

Is it right?
Who can give recommendations on how to use my servers more profitably?

2

You ideally want to have 3 master-eligible nodes in a cluster, so the cluster can continue operating even if 1 master nodes drops. I would therefore recommend the following:

  • Make both data nodes master eligible and set minimum_master_nodes to 2.
  • Make the smaller node a dedicated master node by setting node.ingest to false.
  • Send all traffic directly to the two nodes holding data. Avoid sending any requests through the small dedicated master node.
  • Keep the dedicated ingest node you just added if you feel you need it.