Snapshot to s3 with elasticsearch-cloud-aws


(Andrew Rothstein) #1

Running elasticsearch on my laptop. Trying to create a snapshot using the
elasticsearch-cloud-aws plugin to my "elasticsearch-cre-snapshot" bucket on
s3

andrews-mbp:~ drew$ s3cmd ls | fgrep elasticsearch-cre-snapshot

2014-03-27 04:35 s3://elasticsearch-cre-snapshot

andrews-mbp:~ drew$ curl -XPOST localhost:9200/_snapshot/my_s3_repository
-d '

{

"type": "s3",

"settings": {

    "bucket": "elasticsearch-cre-snapshot",

    "access_key" : "XXXXXX",

    "secret_key" : "YYYYYYYY"

}

}'

{"acknowledged":true}

Loving life. Sh!t is working. Hoorah! Now for the money shot. Run the
snapshot...

andrews-mbp:~ drew$ curl -XGET
localhost:9200/_snapshot/my_s3_repository/snapshot_20140327?wait_for_completion=true

{"error":"SnapshotException[[my_s3_repository:snapshot_20140327] failed to
get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_20140327]]; nested: AmazonS3Exception[Status Code: 404,
AWS Service: Amazon S3, AWS Request ID: EC672767156D1CA8, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.];
","status":500}

What key? 404 what? what? Any suggestions as to what my user error might be?

Here's the versions:

andrews-mbp:~ drew$ curl -s -XGET localhost:9200 | fgrep number

"number" : "1.0.1",

Built the cloud-aws pluging from the es-1.0 branch on github.

andrews-mbp:~ drew$ ls -l
/usr/local/var/lib/elasticsearch/plugins/cloud-aws/

total 28344

-rw-r--r-- 1 drew admin 11934540 Mar 27 00:04 aws-java-sdk-1.7.3.jar

-rw-r--r-- 1 drew admin 58160 Mar 2 00:13 commons-codec-1.4.jar

-rw-r--r-- 1 drew admin 60686 Nov 10 23:00 commons-logging-1.1.1.jar

-rw-r--r-- 1 drew admin 46083 Mar 27 00:19
elasticsearch-cloud-aws-2.0.1-SNAPSHOT.jar

-rw-r--r-- 1 drew admin 424648 Mar 27 00:04 httpclient-4.2.jar

-rw-r--r-- 1 drew admin 223282 Mar 27 00:04 httpcore-4.2.jar

-rw-r--r-- 1 drew admin 34477 Mar 27 00:04
jackson-annotations-2.1.1.jar

-rw-r--r-- 1 drew admin 206379 Mar 27 00:04 jackson-core-2.1.1.jar

-rw-r--r-- 1 drew admin 922643 Mar 27 00:04 jackson-databind-2.1.1.jar

-rw-r--r-- 1 drew admin 581571 Mar 27 00:04 joda-time-2.3.jar
Any suggestions would be greatly appreciated!
-Andrew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ca937821-2936-47a5-b92e-81788ca01668%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Andrew Rothstein) #2

Here's the contents as a gist in the vein of playing by the rules. Any
suggestions for directions shy of adding print statements to the plugin
would be greatly appreciated.


Best regards, Andrew

On Thursday, March 27, 2014 11:15:54 PM UTC-4, Andrew Rothstein wrote:

Running elasticsearch on my laptop. Trying to create a snapshot using the
elasticsearch-cloud-aws plugin to my "elasticsearch-cre-snapshot" bucket on
s3

andrews-mbp:~ drew$ s3cmd ls | fgrep elasticsearch-cre-snapshot

2014-03-27 04:35 s3://elasticsearch-cre-snapshot

andrews-mbp:~ drew$ curl -XPOST localhost:9200/_snapshot/my_s3_repository
-d '

{

"type": "s3",

"settings": {

    "bucket": "elasticsearch-cre-snapshot",

    "access_key" : "XXXXXX",

    "secret_key" : "YYYYYYYY"

}

}'

{"acknowledged":true}

Loving life. Sh!t is working. Hoorah! Now for the money shot. Run the
snapshot...

andrews-mbp:~ drew$ curl -XGET
localhost:9200/_snapshot/my_s3_repository/snapshot_20140327?wait_for_completion=true

{"error":"SnapshotException[[my_s3_repository:snapshot_20140327] failed to
get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_20140327]]; nested: AmazonS3Exception[Status Code: 404,
AWS Service: Amazon S3, AWS Request ID: EC672767156D1CA8, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.];
","status":500}

What key? 404 what? what? Any suggestions as to what my user error might
be?

Here's the versions:

andrews-mbp:~ drew$ curl -s -XGET localhost:9200 | fgrep number

"number" : "1.0.1",

Built the cloud-aws pluging from the es-1.0 branch on github.

andrews-mbp:~ drew$ ls -l
/usr/local/var/lib/elasticsearch/plugins/cloud-aws/

total 28344

-rw-r--r-- 1 drew admin 11934540 Mar 27 00:04 aws-java-sdk-1.7.3.jar

-rw-r--r-- 1 drew admin 58160 Mar 2 00:13 commons-codec-1.4.jar

-rw-r--r-- 1 drew admin 60686 Nov 10 23:00 commons-logging-1.1.1.jar

-rw-r--r-- 1 drew admin 46083 Mar 27 00:19
elasticsearch-cloud-aws-2.0.1-SNAPSHOT.jar

-rw-r--r-- 1 drew admin 424648 Mar 27 00:04 httpclient-4.2.jar

-rw-r--r-- 1 drew admin 223282 Mar 27 00:04 httpcore-4.2.jar

-rw-r--r-- 1 drew admin 34477 Mar 27 00:04
jackson-annotations-2.1.1.jar

-rw-r--r-- 1 drew admin 206379 Mar 27 00:04 jackson-core-2.1.1.jar

-rw-r--r-- 1 drew admin 922643 Mar 27 00:04 jackson-databind-2.1.1.jar

-rw-r--r-- 1 drew admin 581571 Mar 27 00:04 joda-time-2.3.jar
Any suggestions would be greatly appreciated!
-Andrew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/346c532a-6531-4a1d-9329-91f85765206c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Andrew Rothstein) #3

Mystery solved! Use -XPUT to write a snapshot to S3. Not -XGET!

PEBKAC!

On Thursday, March 27, 2014 11:15:54 PM UTC-4, Andrew Rothstein wrote:

Running elasticsearch on my laptop. Trying to create a snapshot using the
elasticsearch-cloud-aws plugin to my "elasticsearch-cre-snapshot" bucket on
s3

andrews-mbp:~ drew$ s3cmd ls | fgrep elasticsearch-cre-snapshot

2014-03-27 04:35 s3://elasticsearch-cre-snapshot

andrews-mbp:~ drew$ curl -XPOST localhost:9200/_snapshot/my_s3_repository
-d '

{

"type": "s3",

"settings": {

    "bucket": "elasticsearch-cre-snapshot",

    "access_key" : "XXXXXX",

    "secret_key" : "YYYYYYYY"

}

}'

{"acknowledged":true}

Loving life. Sh!t is working. Hoorah! Now for the money shot. Run the
snapshot...

andrews-mbp:~ drew$ curl -XGET
localhost:9200/_snapshot/my_s3_repository/snapshot_20140327?wait_for_completion=true

{"error":"SnapshotException[[my_s3_repository:snapshot_20140327] failed to
get snapshots]; nested: IOException[Failed to get
[snapshot-snapshot_20140327]]; nested: AmazonS3Exception[Status Code: 404,
AWS Service: Amazon S3, AWS Request ID: EC672767156D1CA8, AWS Error Code:
NoSuchKey, AWS Error Message: The specified key does not exist.];
","status":500}

What key? 404 what? what? Any suggestions as to what my user error might
be?

Here's the versions:

andrews-mbp:~ drew$ curl -s -XGET localhost:9200 | fgrep number

"number" : "1.0.1",

Built the cloud-aws pluging from the es-1.0 branch on github.

andrews-mbp:~ drew$ ls -l
/usr/local/var/lib/elasticsearch/plugins/cloud-aws/

total 28344

-rw-r--r-- 1 drew admin 11934540 Mar 27 00:04 aws-java-sdk-1.7.3.jar

-rw-r--r-- 1 drew admin 58160 Mar 2 00:13 commons-codec-1.4.jar

-rw-r--r-- 1 drew admin 60686 Nov 10 23:00 commons-logging-1.1.1.jar

-rw-r--r-- 1 drew admin 46083 Mar 27 00:19
elasticsearch-cloud-aws-2.0.1-SNAPSHOT.jar

-rw-r--r-- 1 drew admin 424648 Mar 27 00:04 httpclient-4.2.jar

-rw-r--r-- 1 drew admin 223282 Mar 27 00:04 httpcore-4.2.jar

-rw-r--r-- 1 drew admin 34477 Mar 27 00:04
jackson-annotations-2.1.1.jar

-rw-r--r-- 1 drew admin 206379 Mar 27 00:04 jackson-core-2.1.1.jar

-rw-r--r-- 1 drew admin 922643 Mar 27 00:04 jackson-databind-2.1.1.jar

-rw-r--r-- 1 drew admin 581571 Mar 27 00:04 joda-time-2.3.jar
Any suggestions would be greatly appreciated!
-Andrew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c77b08d5-c4c9-47bb-8e7e-fed7d962233d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4