Hot backup strategy for Elasticsearch

Elastic Stack Elasticsearch
1

Hello List,

Going through the current documentation I found that snapshot/restore
mechanism is one type of backup strategy that we can use for ES clusters.
Any other recommendations?

Using the following

1.elasticsearch-
"version" : {
"number" : "1.3.4",

  1. AWS-cloud-plugin
  2. curator

curator snapshot --repository mys3_repository --all-indices (weekend)
curator snapshot --repository mys3_repository --most-recent 1 (every week
day)

The above would be run as cron jobs from one of the nodes in the cluster.

Let me know of recommendations for hot backup for elastic search cluster.

Thanks,
skm

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fdb9ebae-0352-491c-bca6-dc905cd623ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

No - you should definitely use the snapshot and restore as its the most
stable and efficient way for backups there is.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Oct 15, 2014 at 1:12 AM, skm replysonika@gmail.com wrote:

Hello List,

Going through the current documentation I found that snapshot/restore
mechanism is one type of backup strategy that we can use for ES clusters.
Any other recommendations?

Using the following

1.elasticsearch-
"version" : {
"number" : "1.3.4",

  1. AWS-cloud-plugin
  2. curator

curator snapshot --repository mys3_repository --all-indices (weekend)
curator snapshot --repository mys3_repository --most-recent 1 (every week
day)

The above would be run as cron jobs from one of the nodes in the cluster.

Let me know of recommendations for hot backup for Elasticsearch cluster.

Thanks,
skm

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fdb9ebae-0352-491c-bca6-dc905cd623ae%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fdb9ebae-0352-491c-bca6-dc905cd623ae%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtpT5Q2C2sswPDJRN0KK3xNWGbxdUFoctqGd%3D%2B1q7cs1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thank you for the response!

Usually for large amounts of data (TBs) how the snapshot backup strategy
work? Full snapshots every week and then most-recent snapshots work well?
The most recent would be redundant if there is no new data in the last 24
hrs.?

Thanks,
skm

On Wednesday, October 15, 2014 12:54:13 AM UTC-7, Itamar Syn-Hershko wrote:

No - you should definitely use the snapshot and restore as its the most
stable and efficient way for backups there is.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Oct 15, 2014 at 1:12 AM, skm <reply...@gmail.com <javascript:>>
wrote:

Hello List,

Going through the current documentation I found that snapshot/restore
mechanism is one type of backup strategy that we can use for ES clusters.
Any other recommendations?

Using the following

1.elasticsearch-
"version" : {
"number" : "1.3.4",

  1. AWS-cloud-plugin
  2. curator

curator snapshot --repository mys3_repository --all-indices (weekend)
curator snapshot --repository mys3_repository --most-recent 1 (every week
day)

The above would be run as cron jobs from one of the nodes in the cluster.

Let me know of recommendations for hot backup for Elasticsearch cluster.

Thanks,
skm

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fdb9ebae-0352-491c-bca6-dc905cd623ae%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fdb9ebae-0352-491c-bca6-dc905cd623ae%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e42872cd-7f44-4ada-b1d5-e988edac60e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Incremental. See

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Oct 15, 2014 at 5:15 PM, skm replysonika@gmail.com wrote:

Thank you for the response!

Usually for large amounts of data (TBs) how the snapshot backup strategy
work? Full snapshots every week and then most-recent snapshots work well?
The most recent would be redundant if there is no new data in the last 24
hrs.?

Thanks,
skm

On Wednesday, October 15, 2014 12:54:13 AM UTC-7, Itamar Syn-Hershko wrote:

No - you should definitely use the snapshot and restore as its the most
stable and efficient way for backups there is.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Oct 15, 2014 at 1:12 AM, skm reply...@gmail.com wrote:

Hello List,

Going through the current documentation I found that snapshot/restore
mechanism is one type of backup strategy that we can use for ES clusters.
Any other recommendations?

Using the following

1.elasticsearch-
"version" : {
"number" : "1.3.4",

  1. AWS-cloud-plugin
  2. curator

curator snapshot --repository mys3_repository --all-indices (weekend)
curator snapshot --repository mys3_repository --most-recent 1 (every
week day)

The above would be run as cron jobs from one of the nodes in the cluster.

Let me know of recommendations for hot backup for Elasticsearch cluster.

Thanks,
skm

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/fdb9ebae-0352-491c-bca6-dc905cd623ae%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fdb9ebae-0352-491c-bca6-dc905cd623ae%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e42872cd-7f44-4ada-b1d5-e988edac60e0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e42872cd-7f44-4ada-b1d5-e988edac60e0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtbuP1qpXXDpV-zrkdhuuO6DACq32-LF45z%3DQ9s2QA4Ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Thank you!

I run the snapshot commands couple times and see the same size snapshots in
s3 bucket. Shouldn't it be smaller if this is incremental? there is no
change in the data between these time periods.

Thanks,
skm

On Tuesday, October 14, 2014 3:12:45 PM UTC-7, skm wrote:

Hello List,

Going through the current documentation I found that snapshot/restore
mechanism is one type of backup strategy that we can use for ES clusters.
Any other recommendations?

Using the following

1.elasticsearch-
"version" : {
"number" : "1.3.4",

  1. AWS-cloud-plugin
  2. curator

curator snapshot --repository mys3_repository --all-indices (weekend)
curator snapshot --repository mys3_repository --most-recent 1 (every week
day)

The above would be run as cron jobs from one of the nodes in the cluster.

Let me know of recommendations for hot backup for Elasticsearch cluster.

Thanks,
skm

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3f5f834a-98ed-4f6f-9772-abfdc85eb077%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Incremental means that it has only to copy differences.
As you said, nothing was indexed. So nothing new (or less) was copied.

David

Le 15 oct. 2014 à 17:07, skm replysonika@gmail.com a écrit :

Thank you!

I run the snapshot commands couple times and see the same size snapshots in s3 bucket. Shouldn't it be smaller if this is incremental? there is no change in the data between these time periods.

Thanks,
skm

On Tuesday, October 14, 2014 3:12:45 PM UTC-7, skm wrote:
Hello List,

Going through the current documentation I found that snapshot/restore mechanism is one type of backup strategy that we can use for ES clusters. Any other recommendations?

Using the following

1.elasticsearch-
"version" : {
"number" : "1.3.4",

  1. AWS-cloud-plugin
  2. curator

curator snapshot --repository mys3_repository --all-indices (weekend)
curator snapshot --repository mys3_repository --most-recent 1 (every week day)

The above would be run as cron jobs from one of the nodes in the cluster.

Let me know of recommendations for hot backup for Elasticsearch cluster.

Thanks,
skm

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3f5f834a-98ed-4f6f-9772-abfdc85eb077%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B29432AE-60C8-4407-9CC5-880E9AE424BE%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

7

Thank you!

On Tuesday, October 14, 2014 3:12:45 PM UTC-7, skm wrote:

Hello List,

Going through the current documentation I found that snapshot/restore
mechanism is one type of backup strategy that we can use for ES clusters.
Any other recommendations?

Using the following

1.elasticsearch-
"version" : {
"number" : "1.3.4",

  1. AWS-cloud-plugin
  2. curator

curator snapshot --repository mys3_repository --all-indices (weekend)
curator snapshot --repository mys3_repository --most-recent 1 (every week
day)

The above would be run as cron jobs from one of the nodes in the cluster.

Let me know of recommendations for hot backup for Elasticsearch cluster.

Thanks,
skm

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0703c2b5-e645-4e70-b3bc-f714db48a5c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.