Understanding of recommended ES Cluster Architecture

I am currently working on designing a ELK Domain via AWS ES Service, and I trying to decided how many dedicated master, data, ingest, etc. nodes I require.

I am ingesting data from AWS Cloudtrail via Lambda, bringing in log data across several different accounts, and will be adding log data ingestion from several IIS servers.

Currently, I see near 1,000,000 documents being index per 24 hours. My ELK domain as of yesterday had 3 default nodes with 10GB's of storage each. None dedicated anything.

I am waiting for my ELK to process adding an additional 5 nodes and increase the storage of each node to 30GB's.

I am waiting to see how many logs are to be expected from IIS - though I would assume a good approximation would be another 1,000,000 just to have a marginal overestimation.

I am thinking of going by the rule of thumb and include 3 dedicated master nodes, with the 8 default that currently are being processed, and then I am thinking of including an ingestion node but am not sure what the rule of thumb is for that node type.

Additionally I expect to have around 50 indices.

What is a recommendation for my ELK domain?

Thanks to whomever can assist me!

For that little storage I would recommend going with a basic cluster of 3 identical nodes which all hold data and are master eligible. At this size there is no point adding dedicated node types. How large you need to make the nodes depend on how much storage you need.

Is that 50 indices in total or 50 different time-based indices?

Try to keep the number of indices to a minimum as having lots of small indices and shards in a cluster is very inefficient and can cause performance problems.

Thank you. Just want to confirm 3 nodes total.. correct?

And yes 50 different Indexes. I am indexing based on the source of the log event - i.e. IAM, EC2, S3, etc.

There is often no need to have a separate index per log type so I would recommend you consolidate. Also try adjust the time period covered by each index so you get a shard size ideally over 1GB.

Can you elaborate on your latter statement please?

Please read this blog post.

I understand. If you don't mind taking a look at this so help me understand it better. I am trying to estimate storage requirements. (I have had my ELK taking in data for roughly 3 days and I have pulled up my cluster health and node health).

I just need to know if the disk space used per node is relative to the actual min storage needed per (in this case 3 days) summed together or separately?? The avg source bucket for logs is around 2GB's so I am trying to figure out how these numbers are showing...

Also thanks so much for the help!! :slight_smile:

Another question sorry!! Are shards index dependent? That is to say my consolidated indices I have now would easily procure more than 1GB of data, but another future index for a different source might not. Wondering if that would cause any instability.