Elasticsearch cluster design

Elastic Stack Elasticsearch
1

Hi All,

I'm new on elk stack and planning the design of elasticsearch cluster which needs advice. I have around 9 to 10 million log every day. The log retention is 6 months, 1 year would be the best. I have following question of the cluster design

  1. How much nodes should have? For now, i can have 2 to 3 tb to store log. I plan to have 2 to 3 nodes. 1 node is master and the others are data nodes. The availability of master node down is not my main concern, the cpu and memory usage is the main concern.

  2. Do I need to decrease the amount of Shards and how much would be better in this situation?

  3. If have more than 1 data node, Kibana should point to which data node.

  4. Is it good to place kibana and elasticsearch master into the same node or separate is better.

Thanks

Best Regards,
Peter Chow

2

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.