# Copy index from production to development instance
**URL:** https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960
**Category:** Elasticsearch
**Created:** [June 6, 2014, 5:11pm UTC](https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960 "2014-06-06T17:11:57Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Brian\_Lamb](https://discuss.elastic.co/u/Brian_Lamb)
#### Post date: [June 6, 2014, 5:11pm UTC](https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960/1 "2014-06-06T17:11:57Z")
Hi all,
I want to do a one time copy of the data on my production elastic search
instance to my development elastic search instance. Both are managed by AWS
if that makes this easier. Here is what I tried:
On production:
curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
"type": "s3",
"settings": {
"access\_key": "productionAccessKey",
"bucket": "productionBucketName",
"region": "region",
"secret\_key": "productionSecretKey"
}
}'
curl -XPUT
"[http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02)"
What this does is upload the instance to a production level s3 bucket.
Then in the aws console, I copy all of it to a development level s3 bucket.
Next on development:
curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
"type": "s3",
"settings": {
"access\_key": "developmentAccessKey",
"bucket": "developmentBucketName",
"region": "region",
"secret\_key": "developmentSecretKey"
}
}'
curl -XPOST
"[http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore)"
This gives me the following message:
$ curl -XPOST
"[http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore?pretty=true](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true)"
{
"error" : "SnapshotException[[my\_s3\_repository:snapshot\_2014\_06\_02]
failed to get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot\_2014\_06\_02]]; nested: AmazonS3Exception[Status Code:
404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
"status" : 500
}
Also, when I try to get the snapshots, I get the following:
$ curl -XGET "localhost:9200/\_snapshot/\_status?pretty=true"
{
"snapshots" : []
}
This leads me to believe that I am not connecting the snapshot correctly
but I'm not sure what I am doing incorrectly. Regenerating the index on
development is not really a possibility as it took a few months to generate
the index the first time around. If there is a better way to do this, I'm
all for it.
Thanks,
Brian Lamb
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b5266abf-2ff4-44b3-ba25-734b50d99e83%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b5266abf-2ff4-44b3-ba25-734b50d99e83%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Brian\_Lamb](https://discuss.elastic.co/u/Brian_Lamb)
#### Post date: [June 6, 2014, 5:13pm UTC](https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960/2 "2014-06-06T17:13:06Z")
I should also point out that I had to edit a file in the metadata-snapshot
file to change around the s3 keys and bucket name to match what development
was expecting.
On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:
> Hi all,
>
> I want to do a one time copy of the data on my production Elasticsearch
> instance to my development Elasticsearch instance. Both are managed by AWS
> if that makes this easier. Here is what I tried:
>
> On production:
>
> curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> "type": "s3",
> "settings": {
> "access\_key": "productionAccessKey",
> "bucket": "productionBucketName",
> "region": "region",
> "secret\_key": "productionSecretKey"
> }
> }'
> curl -XPUT "
> [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02)"
>
> What this does is upload the instance to a production level s3 bucket.
>
> Then in the aws console, I copy all of it to a development level s3 bucket.
>
> Next on development:
>
> curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> "type": "s3",
> "settings": {
> "access\_key": "developmentAccessKey",
> "bucket": "developmentBucketName",
> "region": "region",
> "secret\_key": "developmentSecretKey"
> }
> }'
> curl -XPOST "
> [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore)
> "
>
> This gives me the following message:
>
> $ curl -XPOST "
> [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore?pretty=true](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true)
> "
> {
> "error" : "SnapshotException[[my\_s3\_repository:snapshot\_2014\_06\_02]
> failed to get snapshots]; nested: IOException[Failed to get
> [snapshot-snapshot\_2014\_06\_02]]; nested: AmazonS3Exception[Status Code:
> 404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
> NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
> "status" : 500
> }
>
> Also, when I try to get the snapshots, I get the following:
>
> $ curl -XGET "localhost:9200/\_snapshot/\_status?pretty=true"
> {
> "snapshots" :
> }
>
> This leads me to believe that I am not connecting the snapshot correctly
> but I'm not sure what I am doing incorrectly. Regenerating the index on
> development is not really a possibility as it took a few months to generate
> the index the first time around. If there is a better way to do this, I'm
> all for it.
>
> Thanks,
>
> Brian Lamb
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/09fd8162-b39d-4b8d-83bc-c011e4d8bf05%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/09fd8162-b39d-4b8d-83bc-c011e4d8bf05%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Antonio\_Augusto\_Sant](https://discuss.elastic.co/u/Antonio_Augusto_Sant)
#### Post date: [June 6, 2014, 9:13pm UTC](https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960/3 "2014-06-06T21:13:19Z")
Take a look at stream2es [GitHub - elastic/stream2es: Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)](https://github.com/elasticsearch/stream2es)
On Friday, June 6, 2014 2:13:06 PM UTC-3, Brian Lamb wrote:
> I should also point out that I had to edit a file in the metadata-snapshot
> file to change around the s3 keys and bucket name to match what development
> was expecting.
>
> On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:
>
> > Hi all,
> >
> > I want to do a one time copy of the data on my production Elasticsearch
> > instance to my development Elasticsearch instance. Both are managed by AWS
> > if that makes this easier. Here is what I tried:
> >
> > On production:
> >
> > curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> > "type": "s3",
> > "settings": {
> > "access\_key": "productionAccessKey",
> > "bucket": "productionBucketName",
> > "region": "region",
> > "secret\_key": "productionSecretKey"
> > }
> > }'
> > curl -XPUT "
> > [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02)"
> >
> > What this does is upload the instance to a production level s3 bucket.
> >
> > Then in the aws console, I copy all of it to a development level s3
> > bucket.
> >
> > Next on development:
> >
> > curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> > "type": "s3",
> > "settings": {
> > "access\_key": "developmentAccessKey",
> > "bucket": "developmentBucketName",
> > "region": "region",
> > "secret\_key": "developmentSecretKey"
> > }
> > }'
> > curl -XPOST "
> > [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore)
> > "
> >
> > This gives me the following message:
> >
> > $ curl -XPOST "
> > [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore?pretty=true](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true)
> > "
> > {
> > "error" : "SnapshotException[[my\_s3\_repository:snapshot\_2014\_06\_02]
> > failed to get snapshots]; nested: IOException[Failed to get
> > [snapshot-snapshot\_2014\_06\_02]]; nested: AmazonS3Exception[Status Code:
> > 404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
> > NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
> > "status" : 500
> > }
> >
> > Also, when I try to get the snapshots, I get the following:
> >
> > $ curl -XGET "localhost:9200/\_snapshot/\_status?pretty=true"
> > {
> > "snapshots" :
> > }
> >
> > This leads me to believe that I am not connecting the snapshot correctly
> > but I'm not sure what I am doing incorrectly. Regenerating the index on
> > development is not really a possibility as it took a few months to generate
> > the index the first time around. If there is a better way to do this, I'm
> > all for it.
> >
> > Thanks,
> >
> > Brian Lamb
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/52168c96-30ea-4527-b287-676e757b1e6a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/52168c96-30ea-4527-b287-676e757b1e6a%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Brian\_Lamb](https://discuss.elastic.co/u/Brian_Lamb)
#### Post date: [June 26, 2014, 3:45pm UTC](https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960/4 "2014-06-26T15:45:11Z")
Thank you for your suggestion. I tried the stream2es library but I get a
OutOfMemoryError when trying to use that.
On Friday, June 6, 2014 5:13:19 PM UTC-4, Antonio Augusto Santos wrote:
> Take a look at stream2es [GitHub - elastic/stream2es: Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)](https://github.com/elasticsearch/stream2es)
>
> On Friday, June 6, 2014 2:13:06 PM UTC-3, Brian Lamb wrote:
>
> > I should also point out that I had to edit a file in the
> > metadata-snapshot file to change around the s3 keys and bucket name to
> > match what development was expecting.
> >
> > On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:
> >
> > > Hi all,
> > >
> > > I want to do a one time copy of the data on my production Elasticsearch
> > > instance to my development Elasticsearch instance. Both are managed by AWS
> > > if that makes this easier. Here is what I tried:
> > >
> > > On production:
> > >
> > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> > > "type": "s3",
> > > "settings": {
> > > "access\_key": "productionAccessKey",
> > > "bucket": "productionBucketName",
> > > "region": "region",
> > > "secret\_key": "productionSecretKey"
> > > }
> > > }'
> > > curl -XPUT "
> > > [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02)"
> > >
> > > What this does is upload the instance to a production level s3 bucket.
> > >
> > > Then in the aws console, I copy all of it to a development level s3
> > > bucket.
> > >
> > > Next on development:
> > >
> > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> > > "type": "s3",
> > > "settings": {
> > > "access\_key": "developmentAccessKey",
> > > "bucket": "developmentBucketName",
> > > "region": "region",
> > > "secret\_key": "developmentSecretKey"
> > > }
> > > }'
> > > curl -XPOST "
> > > [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore)
> > > "
> > >
> > > This gives me the following message:
> > >
> > > $ curl -XPOST "
> > > [http://localhost:9200/\_snapshot/my\_s3\_repository/snapshot\_2014\_06\_02/\_restore?pretty=true](http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true)
> > > "
> > > {
> > > "error" : "SnapshotException[[my\_s3\_repository:snapshot\_2014\_06\_02]
> > > failed to get snapshots]; nested: IOException[Failed to get
> > > [snapshot-snapshot\_2014\_06\_02]]; nested: AmazonS3Exception[Status Code:
> > > 404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
> > > NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
> > > "status" : 500
> > > }
> > >
> > > Also, when I try to get the snapshots, I get the following:
> > >
> > > $ curl -XGET "localhost:9200/\_snapshot/\_status?pretty=true"
> > > {
> > > "snapshots" :
> > > }
> > >
> > > This leads me to believe that I am not connecting the snapshot correctly
> > > but I'm not sure what I am doing incorrectly. Regenerating the index on
> > > development is not really a possibility as it took a few months to generate
> > > the index the first time around. If there is a better way to do this, I'm
> > > all for it.
> > >
> > > Thanks,
> > >
> > > Brian Lamb
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Himanshu\_Agrawal](https://discuss.elastic.co/u/Himanshu_Agrawal)
#### Post date: [June 26, 2014, 8:15pm UTC](https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960/5 "2014-06-26T20:15:59Z")
From the data you have provided I see that your bucket and keys for
development and production are different. Point your development
elasticsearch instance to the same AWS account and bucket in which you are
storing the snapshot.
On Jun 26, 2014 9:15 PM, "Brian Lamb" [brian.lamb@researchsquare.com](mailto:brian.lamb@researchsquare.com) wrote:
> Thank you for your suggestion. I tried the stream2es library but I get a
> OutOfMemoryError when trying to use that.
>
> On Friday, June 6, 2014 5:13:19 PM UTC-4, Antonio Augusto Santos wrote:
>
> > Take a look at stream2es [GitHub - elastic/stream2es: Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)](https://github.com/elasticsearch/stream2es)
> >
> > On Friday, June 6, 2014 2:13:06 PM UTC-3, Brian Lamb wrote:
> >
> > > I should also point out that I had to edit a file in the
> > > metadata-snapshot file to change around the s3 keys and bucket name to
> > > match what development was expecting.
> > >
> > > On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:
> > >
> > > > Hi all,
> > > >
> > > > I want to do a one time copy of the data on my production elastic
> > > > search instance to my development Elasticsearch instance. Both are managed
> > > > by AWS if that makes this easier. Here is what I tried:
> > > >
> > > > On production:
> > > >
> > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> > > > "type": "s3",
> > > > "settings": {
> > > > "access\_key": "productionAccessKey",
> > > > "bucket": "productionBucketName",
> > > > "region": "region",
> > > > "secret\_key": "productionSecretKey"
> > > > }
> > > > }'
> > > > curl -XPUT "[http://localhost:9200/\_snapshot/my\_s3\_repository/](http://localhost:9200/_snapshot/my_s3_repository/)
> > > > snapshot\_2014\_06\_02"
> > > >
> > > > What this does is upload the instance to a production level s3 bucket.
> > > >
> > > > Then in the aws console, I copy all of it to a development level s3
> > > > bucket.
> > > >
> > > > Next on development:
> > > >
> > > > curl -XPUT '[http://localhost:9200/\_snapshot/my\_s3\_repository](http://localhost:9200/_snapshot/my_s3_repository)' -d '{
> > > > "type": "s3",
> > > > "settings": {
> > > > "access\_key": "developmentAccessKey",
> > > > "bucket": "developmentBucketName",
> > > > "region": "region",
> > > > "secret\_key": "developmentSecretKey"
> > > > }
> > > > }'
> > > > curl -XPOST "[http://localhost:9200/\_snapshot/my\_s3\_repository/](http://localhost:9200/_snapshot/my_s3_repository/)
> > > > snapshot\_2014\_06\_02/\_restore"
> > > >
> > > > This gives me the following message:
> > > >
> > > > $ curl -XPOST "[http://localhost:9200/\_snapshot/my\_s3\_repository/](http://localhost:9200/_snapshot/my_s3_repository/)
> > > > snapshot\_2014\_06\_02/\_restore?pretty=true"
> > > > {
> > > > "error" : "SnapshotException[[my\_s3\_repository:snapshot\_2014\_06\_02]
> > > > failed to get snapshots]; nested: IOException[Failed to get
> > > > [snapshot-snapshot\_2014\_06\_02]]; nested: AmazonS3Exception[Status
> > > > Code: 404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error
> > > > Code: NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
> > > > "status" : 500
> > > > }
> > > >
> > > > Also, when I try to get the snapshots, I get the following:
> > > >
> > > > $ curl -XGET "localhost:9200/\_snapshot/\_status?pretty=true"
> > > > {
> > > > "snapshots" :
> > > > }
> > > >
> > > > This leads me to believe that I am not connecting the snapshot
> > > > correctly but I'm not sure what I am doing incorrectly. Regenerating the
> > > > index on development is not really a possibility as it took a few months to
> > > > generate the index the first time around. If there is a better way to do
> > > > this, I'm all for it.
> > > >
> > > > Thanks,
> > > >
> > > > Brian Lamb
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACTSS%3D6%3DP-Mc%2B4\_o-ROAx0qDtqpRAk4f3VH4OfRSGSbVs61Uwg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACTSS%3D6%3DP-Mc%2B4_o-ROAx0qDtqpRAk4f3VH4OfRSGSbVs61Uwg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:19am UTC](https://discuss.elastic.co/t/copy-index-from-production-to-development-instance/17960/6 "2017-07-06T01:19:26Z")