we are about to deploy EL cluster to our older HW to utilize it and looking for some advices.
My current setup is 8x blade servers with internal storage and 88GB of RAM all together (will be distributed into blades), EL cluster to assume to process around 300mil of time series documents (logs) in total (20mil/month) in approx 160GB of total size. Indexing is not the main issue done in bulk from files, but mainly used for searches/aggregations (Kibana 4)
My plan is to distribute it as follows:
2x master nodes - holding no data, having each 8GB of RAM (16 GB total)
3x workhosre nodes - holding data, having each 16 GB of RAM (48GB total)
1x logstash server (+1 cold backup) - having installed logstash with processing of csv files and seding to EL cluster with 8GB of RAM
Is this setup OK? Any changes to be done?
And three teoretical questions:
- What is better for searches/aggregations - 4 servers with 16GB RAM each or 2 servers with 32GB RAM each?
- Its better to have workhorses the same amount of memory or it doesnt matter at all?
- Master node not holding data should have more memory or leave it all to workhorses ?