Copy index from production to development instance

Elastic Stack Elasticsearch
1

Hi all,

I want to do a one time copy of the data on my production elastic search
instance to my development elastic search instance. Both are managed by AWS
if that makes this easier. Here is what I tried:

On production:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "productionAccessKey",
"bucket": "productionBucketName",
"region": "region",
"secret_key": "productionSecretKey"
}
}'
curl -XPUT
"http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02"

What this does is upload the instance to a production level s3 bucket.

Then in the aws console, I copy all of it to a development level s3 bucket.

Next on development:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "developmentAccessKey",
"bucket": "developmentBucketName",
"region": "region",
"secret_key": "developmentSecretKey"
}
}'
curl -XPOST
"http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore"

This gives me the following message:

$ curl -XPOST
"http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true"
{
"error" : "SnapshotException[[my_s3_repository:snapshot_2014_06_02]
failed to get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_2014_06_02]]; nested: AmazonS3Exception[Status Code:
404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
"status" : 500
}

Also, when I try to get the snapshots, I get the following:

$ curl -XGET "localhost:9200/_snapshot/_status?pretty=true"
{
"snapshots" : [ ]
}

This leads me to believe that I am not connecting the snapshot correctly
but I'm not sure what I am doing incorrectly. Regenerating the index on
development is not really a possibility as it took a few months to generate
the index the first time around. If there is a better way to do this, I'm
all for it.

Thanks,

Brian Lamb

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b5266abf-2ff4-44b3-ba25-734b50d99e83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I should also point out that I had to edit a file in the metadata-snapshot
file to change around the s3 keys and bucket name to match what development
was expecting.

On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:

Hi all,

I want to do a one time copy of the data on my production Elasticsearch
instance to my development Elasticsearch instance. Both are managed by AWS
if that makes this easier. Here is what I tried:

On production:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "productionAccessKey",
"bucket": "productionBucketName",
"region": "region",
"secret_key": "productionSecretKey"
}
}'
curl -XPUT "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02"

What this does is upload the instance to a production level s3 bucket.

Then in the aws console, I copy all of it to a development level s3 bucket.

Next on development:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "developmentAccessKey",
"bucket": "developmentBucketName",
"region": "region",
"secret_key": "developmentSecretKey"
}
}'
curl -XPOST "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore
"

This gives me the following message:

$ curl -XPOST "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true
"
{
"error" : "SnapshotException[[my_s3_repository:snapshot_2014_06_02]
failed to get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_2014_06_02]]; nested: AmazonS3Exception[Status Code:
404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
"status" : 500
}

Also, when I try to get the snapshots, I get the following:

$ curl -XGET "localhost:9200/_snapshot/_status?pretty=true"
{
"snapshots" :
}

This leads me to believe that I am not connecting the snapshot correctly
but I'm not sure what I am doing incorrectly. Regenerating the index on
development is not really a possibility as it took a few months to generate
the index the first time around. If there is a better way to do this, I'm
all for it.

Thanks,

Brian Lamb

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09fd8162-b39d-4b8d-83bc-c011e4d8bf05%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Take a look at stream2es GitHub - elastic/stream2es: Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)

On Friday, June 6, 2014 2:13:06 PM UTC-3, Brian Lamb wrote:

I should also point out that I had to edit a file in the metadata-snapshot
file to change around the s3 keys and bucket name to match what development
was expecting.

On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:

Hi all,

I want to do a one time copy of the data on my production Elasticsearch
instance to my development Elasticsearch instance. Both are managed by AWS
if that makes this easier. Here is what I tried:

On production:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "productionAccessKey",
"bucket": "productionBucketName",
"region": "region",
"secret_key": "productionSecretKey"
}
}'
curl -XPUT "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02"

What this does is upload the instance to a production level s3 bucket.

Then in the aws console, I copy all of it to a development level s3
bucket.

Next on development:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "developmentAccessKey",
"bucket": "developmentBucketName",
"region": "region",
"secret_key": "developmentSecretKey"
}
}'
curl -XPOST "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore
"

This gives me the following message:

$ curl -XPOST "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true
"
{
"error" : "SnapshotException[[my_s3_repository:snapshot_2014_06_02]
failed to get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_2014_06_02]]; nested: AmazonS3Exception[Status Code:
404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
"status" : 500
}

Also, when I try to get the snapshots, I get the following:

$ curl -XGET "localhost:9200/_snapshot/_status?pretty=true"
{
"snapshots" :
}

This leads me to believe that I am not connecting the snapshot correctly
but I'm not sure what I am doing incorrectly. Regenerating the index on
development is not really a possibility as it took a few months to generate
the index the first time around. If there is a better way to do this, I'm
all for it.

Thanks,

Brian Lamb

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52168c96-30ea-4527-b287-676e757b1e6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Thank you for your suggestion. I tried the stream2es library but I get a
OutOfMemoryError when trying to use that.

On Friday, June 6, 2014 5:13:19 PM UTC-4, Antonio Augusto Santos wrote:

Take a look at stream2es GitHub - elastic/stream2es: Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)

On Friday, June 6, 2014 2:13:06 PM UTC-3, Brian Lamb wrote:

I should also point out that I had to edit a file in the
metadata-snapshot file to change around the s3 keys and bucket name to
match what development was expecting.

On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:

Hi all,

I want to do a one time copy of the data on my production Elasticsearch
instance to my development Elasticsearch instance. Both are managed by AWS
if that makes this easier. Here is what I tried:

On production:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "productionAccessKey",
"bucket": "productionBucketName",
"region": "region",
"secret_key": "productionSecretKey"
}
}'
curl -XPUT "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02"

What this does is upload the instance to a production level s3 bucket.

Then in the aws console, I copy all of it to a development level s3
bucket.

Next on development:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "developmentAccessKey",
"bucket": "developmentBucketName",
"region": "region",
"secret_key": "developmentSecretKey"
}
}'
curl -XPOST "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore
"

This gives me the following message:

$ curl -XPOST "
http://localhost:9200/_snapshot/my_s3_repository/snapshot_2014_06_02/_restore?pretty=true
"
{
"error" : "SnapshotException[[my_s3_repository:snapshot_2014_06_02]
failed to get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_2014_06_02]]; nested: AmazonS3Exception[Status Code:
404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
"status" : 500
}

Also, when I try to get the snapshots, I get the following:

$ curl -XGET "localhost:9200/_snapshot/_status?pretty=true"
{
"snapshots" :
}

This leads me to believe that I am not connecting the snapshot correctly
but I'm not sure what I am doing incorrectly. Regenerating the index on
development is not really a possibility as it took a few months to generate
the index the first time around. If there is a better way to do this, I'm
all for it.

Thanks,

Brian Lamb

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

From the data you have provided I see that your bucket and keys for
development and production are different. Point your development
elasticsearch instance to the same AWS account and bucket in which you are
storing the snapshot.
On Jun 26, 2014 9:15 PM, "Brian Lamb" brian.lamb@researchsquare.com wrote:

Thank you for your suggestion. I tried the stream2es library but I get a
OutOfMemoryError when trying to use that.

On Friday, June 6, 2014 5:13:19 PM UTC-4, Antonio Augusto Santos wrote:

Take a look at stream2es GitHub - elastic/stream2es: Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)

On Friday, June 6, 2014 2:13:06 PM UTC-3, Brian Lamb wrote:

I should also point out that I had to edit a file in the
metadata-snapshot file to change around the s3 keys and bucket name to
match what development was expecting.

On Friday, June 6, 2014 1:11:57 PM UTC-4, Brian Lamb wrote:

Hi all,

I want to do a one time copy of the data on my production elastic
search instance to my development Elasticsearch instance. Both are managed
by AWS if that makes this easier. Here is what I tried:

On production:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "productionAccessKey",
"bucket": "productionBucketName",
"region": "region",
"secret_key": "productionSecretKey"
}
}'
curl -XPUT "http://localhost:9200/_snapshot/my_s3_repository/
snapshot_2014_06_02"

What this does is upload the instance to a production level s3 bucket.

Then in the aws console, I copy all of it to a development level s3
bucket.

Next on development:

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
"type": "s3",
"settings": {
"access_key": "developmentAccessKey",
"bucket": "developmentBucketName",
"region": "region",
"secret_key": "developmentSecretKey"
}
}'
curl -XPOST "http://localhost:9200/_snapshot/my_s3_repository/
snapshot_2014_06_02/_restore"

This gives me the following message:

$ curl -XPOST "http://localhost:9200/_snapshot/my_s3_repository/
snapshot_2014_06_02/_restore?pretty=true"
{
"error" : "SnapshotException[[my_s3_repository:snapshot_2014_06_02]
failed to get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_2014_06_02]]; nested: AmazonS3Exception[Status
Code: 404, AWS Service: Amazon S3, AWS Request ID: RequestId, AWS Error
Code: NoSuchKey, AWS Error Message: The specified key does not exist.]; ",
"status" : 500
}

Also, when I try to get the snapshots, I get the following:

$ curl -XGET "localhost:9200/_snapshot/_status?pretty=true"
{
"snapshots" :
}

This leads me to believe that I am not connecting the snapshot
correctly but I'm not sure what I am doing incorrectly. Regenerating the
index on development is not really a possibility as it took a few months to
generate the index the first time around. If there is a better way to do
this, I'm all for it.

Thanks,

Brian Lamb

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0ed279b6-599f-4c90-917a-d377622e12cd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACTSS%3D6%3DP-Mc%2B4_o-ROAx0qDtqpRAk4f3VH4OfRSGSbVs61Uwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.