I have four ELK nodes with about 500GB of data used out of 2TB on each one. These nodes are highly active. As in they are being sent many different logs to them (SQL, ULS, IIS and a lot more). All four of these nodes run everything. As in Elastic Master (three out of four nodes), Elastic Data (all four nodes), Logstash and Kibana. Nodes 01 and 02 have 4 Cores / 28 GB memory and nodes 03 and 04 have 2 Cores / 14 GB memory.
Is it a recommended architecture to split things out? As in have dedicated Master nodes, Data nodes and Logstash nodes? (with a node or two dedicated to Kibana as well) Is this documented? I want to make sure I have an efficient architecture that can sustain for the long run. I understand that Elastic and Logstash are both hungry resource eaters and I am concerned with having them both running on the same VM as each other.
I understand that having everything running on the same VM's might appear to work fine at the beginning, but I suspect over time there would be a serious performance degradation. I am just looking for documentation to support my claims, but unable to find anything like that in the Elastic documentation.