This has likely been discussed many times so apologies in advance for any veteran Elastic members! As someone reasonably new to the Stack (despite being a forum member for a while now). I'd like to get some advice before committing to he purchases of new hardware. We currently have a small "proof of concept" setup which involves:
1x "Hot" data node - 10 Core Intel Gold CPU - 64GB RAM - SSD storage ~12TB 1x "Warm" data node - 10 Core Intel Gold CPU - 64GB RAM - HDD Storage ~48TB 1x Logstash server - 10 Core Intel Gold CPU - 32GB RAM - SSD Storage 1x Kibana LXC container - 4GB RAM - Shared Storage
I'd like to scale this out for production to look like:
3x "Hot" data node - 10 Core Intel Gold CPU - 64GB RAM - SSD storage ~12TB 3x "Warm" data node - 10 Core Intel Gold CPU - 64GB RAM - HDD Storage ~48TB 3x Master nodes server - Quad Core Intel CPU - 16GB RAM - HDD Storage 2x Logstash server - 10 Core Intel Gold CPU - 32GB RAM - SSD Storage
My first question is this; Does the above look appropriate for a cluster ingesting around 20-30GB per day, with the view to expanding this to include system metrics, audit logs and heartbeat checks? I appreciate this is a difficult question to answer so I'm approaching this from the perspective of data ingestion rate vs hardware resource rather than storage capacity.
Secondly I am considering adding 3x Coordinator nodes which will also run Kibana pointed to localhost. These will sit behind HAProxy loadbalancers to provide load balancing and high availability.
My understanding is that this will allow the "Scatter" and "Gather" phase to be performed on the coordinator nodes rather than a nominated data node. Am I correct in this assumption that Kibana would only be able to point to a single data node to perform queries without the coordinator?
My final question is this; what kind of resources should I be looking at for the coordinator nodes in relating to the rest of the stack? My thinking was something slightly more powerful that the Master nodes but not as "beefy" as the data nodes.
Appreciate any feedback