# Understanding of recommended ES Cluster Architecture **URL:** \ **Category:** Elasticsearch\ **Created:** [July 30, 2020, 2:52pm UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243 "2020-07-30T14:52:23Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![neldreth](https://avatars.discourse-cdn.com/v4/letter/n/a88e4f/32.png) [@neldreth](https://discuss.elastic.co/u/neldreth)\ **Post date:** [July 30, 2020, 2:52pm UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243/1 "2020-07-30T14:52:24Z")
I am currently working on designing a ELK Domain via AWS ES Service, and I trying to decided how many dedicated master, data, ingest, etc. nodes I require. I am ingesting data from AWS Cloudtrail via Lambda, bringing in log data across several different accounts, and will be adding log data ingestion from several IIS servers. Currently, I see near 1,000,000 documents being index per 24 hours. My ELK domain as of yesterday had 3 default nodes with 10GB's of storage each. None dedicated anything. I am waiting for my ELK to process adding an additional 5 nodes and increase the storage of each node to 30GB's. I am waiting to see how many logs are to be expected from IIS - though I would assume a good approximation would be another 1,000,000 just to have a marginal overestimation. I am thinking of going by the rule of thumb and include 3 dedicated master nodes, with the 8 default that currently are being processed, and then I am thinking of including an ingestion node but am not sure what the rule of thumb is for that node type. Additionally I expect to have around 50 indices. What is a recommendation for my ELK domain? Thanks to whomever can assist me! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 30, 2020, 6:45pm UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243/2 "2020-07-30T18:45:46Z")
For that little storage I would recommend going with a basic cluster of 3 identical nodes which all hold data and are master eligible. At this size there is no point adding dedicated node types. How large you need to make the nodes depend on how much storage you need. > [@neldreth](#): > > Additionally I expect to have around 50 indices. Is that 50 indices in total or 50 different time-based indices? Try to keep the number of indices to a minimum as having lots of small indices and shards in a cluster is very inefficient and can cause performance problems. ---
**Author:** ![neldreth](https://avatars.discourse-cdn.com/v4/letter/n/a88e4f/32.png) [@neldreth](https://discuss.elastic.co/u/neldreth)\ **Post date:** [July 31, 2020, 5:26pm UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243/3 "2020-07-31T17:26:37Z")
Thank you. Just want to confirm 3 nodes total.. correct? And yes 50 different Indexes. I am indexing based on the source of the log event - i.e. IAM, EC2, S3, etc. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 31, 2020, 5:31pm UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243/4 "2020-07-31T17:31:30Z")
There is often no need to have a separate index per log type so I would recommend you consolidate. Also try adjust the time period covered by each index so you get a shard size ideally over 1GB. ---
**Author:** ![neldreth](https://avatars.discourse-cdn.com/v4/letter/n/a88e4f/32.png) [@neldreth](https://discuss.elastic.co/u/neldreth)\ **Post date:** [July 31, 2020, 5:45pm UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243/5 "2020-07-31T17:45:10Z")
Can you elaborate on your latter statement please? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 31, 2020, 5:47pm UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243/6 "2020-07-31T17:47:21Z")
Please read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2020, 11:51am UTC](https://discuss.elastic.co/t/understanding-of-recommended-es-cluster-architecture/243243/8 "2020-09-03T11:51:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.