Elastic search - exporting index from existing cluster to a different cluster

Elastic Stack Elasticsearch
1

Can I use the snapshot and restore module of elastic search to export one
index to all together to a new elastic search cluster.

For e.g. Can I export all the indices from development cluster to QA
cluster?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Yes

--
David Pilato | Technical Advocate | elasticsearch.com
david.pilato@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 22 septembre 2014 à 12:29:44, Vijay Tiwary (vijaykr.tiwary@gmail.com) a écrit:

Can I use the snapshot and restore module of elastic search to export one index to all together to a new elastic search cluster.

For e.g. Can I export all the indices from development cluster to QA cluster?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.541ffa32.5bd062c2.825%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

3

Thanks David.

So suppose, I have two cluster "dev cluster" and "QA cluster" and trying to
copy all the indices from the dev to QA cluster.

  1. Create a repository in dev
  2. snapshot all the indices
  3. Create a repository in QA
  4. copy all the files from dev repository to QA
  5. Restore the indices from the QA snapshot in QA cluster

Please let me know whether I export the indices by following the above
steps. If yes, then is this the only way or there are more easier and
better approach.

On Monday, September 22, 2014 4:00:19 PM UTC+5:30, David Pilato wrote:

Yes

--
David Pilato | Technical Advocate | elasticsearch.com
http://elasticsearch.com
david....@elasticsearch.com <javascript:>
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
http://twitter.com/scrutmydocs
https://twitter.com/scrutmydocs

Le 22 septembre 2014 à 12:29:44, Vijay Tiwary (vijaykr...@gmail.com
<javascript:>) a écrit:

Can I use the snapshot and restore module of Elasticsearch to export
one index to all together to a new Elasticsearch cluster.

For e.g. Can I export all the indices from development cluster to QA
cluster?

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6978ea9b-54b1-41f1-85bc-e4f5fc86d19c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Exactly.

Note that if you can simply « mount » your repository FS on both systems, you won’t need to run step 4.
Same applies if you are using S3, Azure or HDFS.

--
David Pilato | Technical Advocate | elasticsearch.com
david.pilato@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 22 septembre 2014 à 13:12:35, Vijay Tiwary (vijaykr.tiwary@gmail.com) a écrit:

Thanks David.

So suppose, I have two cluster "dev cluster" and "QA cluster" and trying to copy all the indices from the dev to QA cluster.

  1. Create a repository in dev
  2. snapshot all the indices
  3. Create a repository in QA
  4. copy all the files from dev repository to QA
  5. Restore the indices from the QA snapshot in QA cluster

Please let me know whether I export the indices by following the above steps. If yes, then is this the only way or there are more easier and better approach.

On Monday, September 22, 2014 4:00:19 PM UTC+5:30, David Pilato wrote:
Yes

--
David Pilato | Technical Advocate | elasticsearch.com
david....@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 22 septembre 2014 à 12:29:44, Vijay Tiwary (vijaykr...@gmail.com) a écrit:

Can I use the snapshot and restore module of elastic search to export one index to all together to a new elastic search cluster.

For e.g. Can I export all the indices from development cluster to QA cluster?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6978ea9b-54b1-41f1-85bc-e4f5fc86d19c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.54200bee.3352255a.825%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

5

I am able to create the snapshot however when i am trying to restore the
sanpshot in a different cluster then the indexes are getting created
however there is no data in the index

On Monday, September 22, 2014 5:16:02 PM UTC+5:30, David Pilato wrote:

Exactly.

Note that if you can simply « mount » your repository FS on both systems,
you won’t need to run step 4.
Same applies if you are using S3, Azure or HDFS.

--
David Pilato | Technical Advocate | elasticsearch.com
http://elasticsearch.com
david....@elasticsearch.com <javascript:>
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
http://twitter.com/scrutmydocs
https://twitter.com/scrutmydocs

Le 22 septembre 2014 à 13:12:35, Vijay Tiwary (vijaykr...@gmail.com
<javascript:>) a écrit:

Thanks David.

So suppose, I have two cluster "dev cluster" and "QA cluster" and trying
to copy all the indices from the dev to QA cluster.

  1. Create a repository in dev
  2. snapshot all the indices
  3. Create a repository in QA
  4. copy all the files from dev repository to QA
  5. Restore the indices from the QA snapshot in QA cluster

Please let me know whether I export the indices by following the above
steps. If yes, then is this the only way or there are more easier and
better approach.

On Monday, September 22, 2014 4:00:19 PM UTC+5:30, David Pilato wrote:

Yes

--

Le 22 septembre 2014 à 12:29:44, Vijay Tiwary (vijaykr...@gmail.com) a
écrit:

Can I use the snapshot and restore module of Elasticsearch to export
one index to all together to a new Elasticsearch cluster.

For e.g. Can I export all the indices from development cluster to QA
cluster?

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6978ea9b-54b1-41f1-85bc-e4f5fc86d19c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6978ea9b-54b1-41f1-85bc-e4f5fc86d19c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/55d25e2f-a235-444a-a6b3-12fbc53d84a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Sorry. I don't understand what you mean.

So snapshot works?
Then you copied files to another place.
The you restored?

And what happened?

--
David Pilato | Technical Advocate | elasticsearch.com
david.pilato@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 8 octobre 2014 à 14:22:09, Vijay Tiwary (vijaykr.tiwary@gmail.com) a écrit:

I am able to create the snapshot however when i am trying to restore the sanpshot in a different cluster then the indexes are getting created however there is no data in the index

On Monday, September 22, 2014 5:16:02 PM UTC+5:30, David Pilato wrote:
Exactly.

Note that if you can simply « mount » your repository FS on both systems, you won’t need to run step 4.
Same applies if you are using S3, Azure or HDFS.

--
David Pilato | Technical Advocate | elasticsearch.com
david....@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 22 septembre 2014 à 13:12:35, Vijay Tiwary (vijaykr...@gmail.com) a écrit:

Thanks David.

So suppose, I have two cluster "dev cluster" and "QA cluster" and trying to copy all the indices from the dev to QA cluster.

  1. Create a repository in dev
  2. snapshot all the indices
  3. Create a repository in QA
  4. copy all the files from dev repository to QA
  5. Restore the indices from the QA snapshot in QA cluster

Please let me know whether I export the indices by following the above steps. If yes, then is this the only way or there are more easier and better approach.

On Monday, September 22, 2014 4:00:19 PM UTC+5:30, David Pilato wrote:
Yes

--
David Pilato | Technical Advocate | elasticsearch.com
david....@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 22 septembre 2014 à 12:29:44, Vijay Tiwary (vijaykr...@gmail.com) a écrit:

Can I use the snapshot and restore module of elastic search to export one index to all together to a new elastic search cluster.

For e.g. Can I export all the indices from development cluster to QA cluster?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4af4e741-0fa8-4d9a-88d4-b353480b0fde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6978ea9b-54b1-41f1-85bc-e4f5fc86d19c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/55d25e2f-a235-444a-a6b3-12fbc53d84a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.54355644.25e45d32.bc27%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

7

We run into a similar problems recently after a very recent upgrade from ES
1.2.2 to ES 1.3.4.

In our scenario, we have two ES clusters and a HDFS system which both can
access.

Indices are created on one of them, snapshotted to HDFS repository (named
after the index) and then restored on the other cluster. This worked fine
(when ignoring random error messages) on 1.2.2. After the upgrade to 1.3.4,
however, clusters don't seem to be able to see new snapshots made on the
other system. Files sit in HDFS, it is possible to restore locally,
accessing:
https://..com/_snapshot/<repository_name>/_all
returns the expected list while
https://..com/_snapshot/<repository_name>/_all
is empty. Beforehand, the lists were identical.

I will try to check version 1.3.2 tomorrow, but remaining clueless at the
moment.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f241a3aa-fd8d-486c-8f6b-40faa701bccd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.