Hi. I'm an intern to a company and one of the managers asks me to evaluate Elastic Stack as an possible option for their use to ease the hassle of analyzing logs. I'm new to Elastic Stack and to the whole log analysis and DevOps environment as I am still studying. He was asking for me to setup a standalone system initially for testing but then he asks me some question about the cluster version instead.
Maybe most of you struck this question as dumb but I did research a lot and got mixed answer for it. So, as the title mentioned, the general question is what is the difference between the two? I put in the documentation I done like the cluster is better cause you can centralize the logs processed, the speed is better, scalable etc but he want more.
The use case is that each analyst (there's about half of dozen) will have their own Elastic Stack and can analyse their assigned logs.
The question he give me are as below;
- When is the suitable time to use standalone and cluster?
- What are distinct benefit of cluster over standalone?
- what is nodes and what are the basic requirements of this different kind of nodes (ES, Logstash, Kibana)?
- What is sharding in ES actually and how to use it?
- How to tell when is the right time to add a node and what is the calculation for an optimum infrastructure?
Now, this is my question for you kind people in here;
- Is there any credible source other than the official documentation that details the basis of cluster infrastructure and is there any popular topology that is proven?
- I know from documentation that a shard in ES is ~50GB for max capacity. But the question is how to actually calculate the ratio of primary and replica shard towards the GB of logs and is there a proctical limit to the shards itself?
- How to effectively maximize performance while not entirely making the machine seppuku itself?
- Is there strategies of using cluster-size Elastic Stack and how t odeploy it?
- How can I get really good to do manage Elastic Stack so that I can explain in depth about this to my manager other than literally reading it from the official documentation?
Sorry for the very long questionnaire and I'm really appreciate anyone willing to help me with this 