Handling growing backup repository size greater than 10TB?


#1

Hi,

--We are using ELK stack for real time analytics with each day's data being about 100GB in size (and growing). Overall cluster size for 10 days data (with replication factor=1) is 1TB.
-- I assume the snapshots taken with curator are incremental backups, since I see a single directory for each unique index
--My doubt is about what do we do when the backup NFS mount gets filled up. Since it is curator backups, of all snapshots taken daily, it has already reached 5TB for 1 month's data. Our mount is of size 10TB. Once it fills up, should I move entire data to some other mount and continue with current mount. For example, if I copy over the back up to some other place and start over again, and if a index happens to be in both backups, how does restore work ?

Any help is much appreciated.

More information:
--We take daily backups like this:

/usr/bin/curator --host 'es_host' --port 9200  snapshot  --repository rtp_es_backup_10TB  indices  --all-indices
 /usr/bin/curator --host 'es_host' --port 9200  optimize indices --older-than 2  --time-unit days --timestring '%Y.%m.%d' 
 /usr/bin/curator --host 'es_host' --port 9200   delete indices --older-than 10 --time-unit days --timestring '%Y.%m.\%d'  

This the repository structure created by curator (sample snippets)

ls es-backup

-rw-r--r-- 1 xxx yyy   1010 Sep 22 02:00 metadata-curator-20150921203002
drwx------ 2 xxx yyy   327680 Sep 22 02:00 indices
-rw-r--r-- 1 xxx yyy   1056 Sep 22 02:46 snapshot-curator-20150921203002

ls es-backup/indices

drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.15
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.16
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.17
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.18
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.19
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.20
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.21
drwxr-xr-x 2 xxx yyy 64   Sep 22 02:44 content-2015.09.22

ls es-backup/indices/content-2015.09.17

-rw-r--r-- 1  xxx yyy    806 Sep 17 02:00 snapshot-curator-20150916203003
-rw-r--r-- 1  xxx yyy    806 Sep 18 02:00 snapshot-curator-20150917203003
-rw-r--r-- 1  xxx yyy    806 Sep 19 02:00 snapshot-curator-20150918203003
-rw-r--r-- 1  xxx yyy    806 Sep 20 02:00 snapshot-curator-20150919203002
-rw-r--r-- 1  xxx yyy    806 Sep 21 02:00 snapshot-curator-20150920203002
-rw-r--r-- 1  xxx yyy    808 Sep 22 02:00 snapshot-curator-20150921203002
drwxr-xr-x 2  xxx yyy   8192 Sep 22 02:03 2
drwxr-xr-x 2  xxx yyy   8192 Sep 22 02:03 1
drwxr-xr-x 2  xxx yyy  12288 Sep 22 02:04 3
drwxr-xr-x 2  xxx yyy  12288 Sep 22 02:06 4
drwxr-xr-x 2  xxx yyy  12288 Sep 22 02:13 0

Thanks
Preetam


(system) #2