I am currently working on designing a ELK Domain via AWS ES Service, and I trying to decided how many dedicated master, data, ingest, etc. nodes I require.
I am ingesting data from AWS Cloudtrail via Lambda, bringing in log data across several different accounts, and will be adding log data ingestion from several IIS servers.
Currently, I see near 1,000,000 documents being index per 24 hours. My ELK domain as of yesterday had 3 default nodes with 10GB's of storage each. None dedicated anything.
I am waiting for my ELK to process adding an additional 5 nodes and increase the storage of each node to 30GB's.
I am waiting to see how many logs are to be expected from IIS - though I would assume a good approximation would be another 1,000,000 just to have a marginal overestimation.
I am thinking of going by the rule of thumb and include 3 dedicated master nodes, with the 8 default that currently are being processed, and then I am thinking of including an ingestion node but am not sure what the rule of thumb is for that node type.
Additionally I expect to have around 50 indices.
What is a recommendation for my ELK domain?
Thanks to whomever can assist me!