Hi,
--We are using ELK stack for real time analytics with each day's data being about 100GB in size (and growing). Overall cluster size for 10 days data (with replication factor=1) is 1TB.
-- I assume the snapshots taken with curator are incremental backups, since I see a single directory for each unique index
--My doubt is about what do we do when the backup NFS mount gets filled up. Since it is curator backups, of all snapshots taken daily, it has already reached 5TB for 1 month's data. Our mount is of size 10TB. Once it fills up, should I move entire data to some other mount and continue with current mount. For example, if I copy over the back up to some other place and start over again, and if a index happens to be in both backups, how does restore work ?
Any help is much appreciated.
More information:
--We take daily backups like this:
/usr/bin/curator --host 'es_host' --port 9200 snapshot --repository rtp_es_backup_10TB indices --all-indices
/usr/bin/curator --host 'es_host' --port 9200 optimize indices --older-than 2 --time-unit days --timestring '%Y.%m.%d'
/usr/bin/curator --host 'es_host' --port 9200 delete indices --older-than 10 --time-unit days --timestring '%Y.%m.\%d'
This the repository structure created by curator (sample snippets)
ls es-backup
-rw-r--r-- 1 xxx yyy 1010 Sep 22 02:00 metadata-curator-20150921203002
drwx------ 2 xxx yyy 327680 Sep 22 02:00 indices
-rw-r--r-- 1 xxx yyy 1056 Sep 22 02:46 snapshot-curator-20150921203002
ls es-backup/indices
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.15
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.16
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.17
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.18
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.19
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.20
drwxr-xr-x 2 xxx yyy 4096 Sep 22 02:00 content-2015.09.21
drwxr-xr-x 2 xxx yyy 64 Sep 22 02:44 content-2015.09.22
ls es-backup/indices/content-2015.09.17
-rw-r--r-- 1 xxx yyy 806 Sep 17 02:00 snapshot-curator-20150916203003
-rw-r--r-- 1 xxx yyy 806 Sep 18 02:00 snapshot-curator-20150917203003
-rw-r--r-- 1 xxx yyy 806 Sep 19 02:00 snapshot-curator-20150918203003
-rw-r--r-- 1 xxx yyy 806 Sep 20 02:00 snapshot-curator-20150919203002
-rw-r--r-- 1 xxx yyy 806 Sep 21 02:00 snapshot-curator-20150920203002
-rw-r--r-- 1 xxx yyy 808 Sep 22 02:00 snapshot-curator-20150921203002
drwxr-xr-x 2 xxx yyy 8192 Sep 22 02:03 2
drwxr-xr-x 2 xxx yyy 8192 Sep 22 02:03 1
drwxr-xr-x 2 xxx yyy 12288 Sep 22 02:04 3
drwxr-xr-x 2 xxx yyy 12288 Sep 22 02:06 4
drwxr-xr-x 2 xxx yyy 12288 Sep 22 02:13 0
Thanks
Preetam