Restore multiple months of data


(Ed) #1

Hi, I have been requested to restore multiple months of data for people search historical log files.

I have been backing up my elasticsearch indexes for +1 year to S3 via the backup/restore API in Elastic

I create an index every day which is ~160GB each day. with 5 shards

So If I wanted to restore 30 days - that would be ~6TB of data per month and 150 non-replicated shards

any advice before I create AWS servers to handle this short time need? My thoughts are to do the following

    1. can't go with Elastic Cloud as purchasing takes a while in my company.
    1. AWS elasticsearch is limited to 30gb which means I would have spawn up 5 instances for each day
    1. Since my install is all ansible'ized I was going to build my own AWS EC2 cluster ~10 Servers/nodes per month with ~600gb storage and select a maybe a medium size instance, and then start restoring and adding servers as needed

(David Pilato) #2

can't go with Elastic Cloud as purchasing takes a while in my company.

Did you see https://aws.amazon.com/marketplace/pp/B01N6YCISK ?
You can subscribe to elastic cloud using your AWS account.


(Ed) #3

I did not know Elastic Cloud could do this, that is now an option :slight_smile: I will review that.

see always good to ask :slight_smile:


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.