[ES-1.4.2] Snapshot FS Repository directory structure

Elastic Stack Elasticsearch
1

Hi All,
We're using the snapshot API to backup our ES cluster and as time advances,
the repository storage utilization is growing.
Currently I use 'rsync' to copy snapshot repository data outside of our
production environment, but eventually I'd need to delete some snapshots to
free space for new ones.

The repository keeps files like 'index', 'snapshot-*' and 'metadata-' files
and I suspect that deleting these files will fail the repository from being
functional.

Is there a way that i can 'rsync' the snapshot repository to another
location and delete the existing snapshots?
Can I delete that snapshot data itself (the 'indices' directory the resides
in the repository) without damaging the repository itself?

Thanks for any idea,
Yarden

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

In these sorts of cases it may make sense to have a repo for a week/months
worth of snapshots, then just retain however much you need in these rolling
repos. Kind of like what LS does for its indices.

Regarding deleting the files, take a look at

On 16 February 2015 at 19:04, Yarden Bar yarden@astoncapitalmanagement.com
wrote:

Hi All,
We're using the snapshot API to backup our ES cluster and as time
advances, the repository storage utilization is growing.
Currently I use 'rsync' to copy snapshot repository data outside of our
production environment, but eventually I'd need to delete some snapshots to
free space for new ones.

The repository keeps files like 'index', 'snapshot-*' and 'metadata-'
files and I suspect that deleting these files will fail the repository from
being functional.

Is there a way that i can 'rsync' the snapshot repository to another
location and delete the existing snapshots?
Can I delete that snapshot data itself (the 'indices' directory the
resides in the repository) without damaging the repository itself?

Thanks for any idea,
Yarden

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_gCbDhBwyGWRrHR42REy_4pti60psX73ue9Z5rKSC3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

If I understand you correctly, the monthly/weekly repository will get
purged when we decide to, right?

Another solution crossed my mind:
Thinking of HDFS, which I'm not so familiar with, but is there some kind of
HDFS node which is similar to ES node which is not a master nor a data?
This way all snapshot will go through a HDFS 'gateway' node to outside of
our prod env.

On Monday, February 16, 2015 at 11:30:14 AM UTC+2, Mark Walkom wrote:

In these sorts of cases it may make sense to have a repo for a week/months
worth of snapshots, then just retain however much you need in these rolling
repos. Kind of like what LS does for its indices.

Regarding deleting the files, take a look at
Elasticsearch Platform — Find real-time answers at scale | Elastic

On 16 February 2015 at 19:04, Yarden Bar <
yar...@astoncapitalmanagement.com <javascript:>> wrote:

Hi All,
We're using the snapshot API to backup our ES cluster and as time
advances, the repository storage utilization is growing.
Currently I use 'rsync' to copy snapshot repository data outside of our
production environment, but eventually I'd need to delete some snapshots to
free space for new ones.

The repository keeps files like 'index', 'snapshot-*' and 'metadata-'
files and I suspect that deleting these files will fail the repository from
being functional.

Is there a way that i can 'rsync' the snapshot repository to another
location and delete the existing snapshots?
Can I delete that snapshot data itself (the 'indices' directory the
resides in the repository) without damaging the repository itself?

Thanks for any idea,
Yarden

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3d7428c8-2245-4ba1-9109-46c8877f1913%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.