Hi, I have been requested to restore multiple months of data for people search historical log files.
I have been backing up my elasticsearch indexes for +1 year to S3 via the backup/restore API in Elastic
I create an index every day which is ~160GB each day. with 5 shards
So If I wanted to restore 30 days - that would be ~6TB of data per month and 150 non-replicated shards
any advice before I create AWS servers to handle this short time need? My thoughts are to do the following
- can't go with Elastic Cloud as purchasing takes a while in my company.
- AWS elasticsearch is limited to 30gb which means I would have spawn up 5 instances for each day
- Since my install is all ansible'ized I was going to build my own AWS EC2 cluster ~10 Servers/nodes per month with ~600gb storage and select a maybe a medium size instance, and then start restoring and adding servers as needed