Got a log cluster with 3 nodes (all data, ingest, master nodes) with 64GB ram, 1TB SSD and E5-26xxvx's. The load is between 30 and 70% depending on the version of the CPU. All running Windows Server.
Some specs of the cluster
- Approx 1.500.000.000 docs
- 6500 shards
- 5-7 index patterns
- Index rate is 300-400/s (primary shards)
- 1 replica/5 shards pr index
- Search rate below 100/s
Looking to upgrade the cluster with 1 or 2 nodes in order to:
- Increase storage, due to more applications adding data.
- Improve search response time from Kibana and Grafana. Tend to get time outs in Kibana and just raised the timeout to 60s. Grafana will be used a lot more, so search rate will increase.
My question is whether I should change the cluster to have dedicated data and master nodes (e.g. I have the possibility to add vm's if required) or if I should just add the extra nodes as data, master and ingest nodes like the current setup? Any advice?