AmazonClientException[Unable to unmarshall error response...] when creating snapshot in S3

Elastic Stack Elasticsearch
1

I`ve set up ES v.1.1.1 + AWS-cloud-plugin 2.1.1 on an EC2 instance, using
a Role to give access to my S3 bucket. Everything seems fine, I can put,
list and get objects from inside the instance (tested with node.js code),
but when I try to create an ES snapshot I get the following error message:

$ curl -XPUT
"localhost:9200/_snapshot/my_snapshot_repo/snapshot1?wait_for_completion=true"
{"error":"SnapshotCreationException[[my_snapshot_repo:snapshot1] failed to
create snapshot]; nested: IOException[Failed to get [snapshot-snapshot1]];
nested: AmazonClientException[Unable to unmarshall error response (The
declaration for the entity "ContentType" must end with '>'.). Response
Code: 417, Response Text: Expectation Failed]; nested:
SAXParseException[The declaration for the entity "ContentType" must end
with '>'.]; ","status":500}

Full error stack here: https://gist.github.com/anonymous/11304863

I`ve set up my snapshot repository with:

$ curl -XPUT 'http://localhost:9200/_snapshot/my_snapshot_repo' -d '{
"type": "s3", "settings": { "bucket": "mybucket","region": "sa-east"}}'

My elasticsearch.yml aws config is as follows:

...
cluster.name: montadores
cloud.aws.region: sa-east-1
discovery:
type: ec2

discovery.ec2.tag.elasticsearch_cluster: mobile_montadores
cloud.node.auto_attributes: true
...

Discovery seems OK, as I only have 1 instance running:

...

[2014-04-25 21:16:53,265][INFO ][cluster.service ] [Cap 'N Hawk]
new_master [Cap 'N
Hawk][M013xeCxQSu8QbZzdNucVA][ip-10-152-89-168][inet[/10.152.89.168:9300]]{aws_availability_zone=sa-east-1a},
reason: zen-disco-join (elected_as_master)

...

[2014-04-25 21:16:56,005][INFO ][node ] [Cap 'N Hawk]
started

The only similar thing I found to this error was an old closed
issue: https://github.com/elasticsearch/elasticsearch/issues/1137

Can anybody shed some light if I`m doing something wrong?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/112aa619-e4db-4dc7-9a30-c86e8c0d2414%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Paulo,

Did you ever figure out your error? I just encountered the exact same
problem and was hoping you had found a solution

Thanks,
Pete

On Friday, April 25, 2014 6:14:38 PM UTC-4, Paulo Correa wrote:

I`ve set up ES v.1.1.1 + AWS-cloud-plugin 2.1.1 on an EC2 instance, using
a Role to give access to my S3 bucket. Everything seems fine, I can put,
list and get objects from inside the instance (tested with node.js code),
but when I try to create an ES snapshot I get the following error message:

$ curl -XPUT
"localhost:9200/_snapshot/my_snapshot_repo/snapshot1?wait_for_completion=true"
{"error":"SnapshotCreationException[[my_snapshot_repo:snapshot1] failed to
create snapshot]; nested: IOException[Failed to get [snapshot-snapshot1]];
nested: AmazonClientException[Unable to unmarshall error response (The
declaration for the entity "ContentType" must end with '>'.). Response
Code: 417, Response Text: Expectation Failed]; nested:
SAXParseException[The declaration for the entity "ContentType" must end
with '>'.]; ","status":500}

Full error stack here: elasticsearch log error · GitHub

I`ve set up my snapshot repository with:

$ curl -XPUT 'http://localhost:9200/_snapshot/my_snapshot_repo' -d '{
"type": "s3", "settings": { "bucket": "mybucket","region": "sa-east"}}'

My elasticsearch.yml aws config is as follows:

...
cluster.name: montadores
cloud.aws.region: sa-east-1
discovery:
type: ec2

discovery.ec2.tag.elasticsearch_cluster: mobile_montadores
cloud.node.auto_attributes: true
...

Discovery seems OK, as I only have 1 instance running:

...

[2014-04-25 21:16:53,265][INFO ][cluster.service ] [Cap 'N Hawk]
new_master [Cap 'N
Hawk][M013xeCxQSu8QbZzdNucVA][ip-10-152-89-168][inet[/10.152.89.168:9300]]{aws_availability_zone=sa-east-1a},
reason: zen-disco-join (elected_as_master)

...

[2014-04-25 21:16:56,005][INFO ][node ] [Cap 'N Hawk]
started

The only similar thing I found to this error was an old closed issue:
Snapshotting to S3 occasionally fails · Issue #1137 · elastic/elasticsearch · GitHub

Can anybody shed some light if I`m doing something wrong?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f86f89fc-6517-4e1b-ae6b-bce9e93d24e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Pete,

we ran some more tests to see if we could narrow down the problem. We found
out that whenever we had an instance that was not in AWS' default-vpc
launch an snapshot, the problem occurred. That was the only factor that
made the problem happen, so it is not an issue with permissions, role,
buckets, etc, but rather may be a routing problem.

We're thinking of submitting a ticket to AWS Support, as we're still facing
the error when we launch an instance in our internal VPC.

Em quinta-feira, 8 de maio de 2014 10h21min48s UTC-3, Pete Michel escreveu:

Paulo,

Did you ever figure out your error? I just encountered the exact same
problem and was hoping you had found a solution

Thanks,
Pete

On Friday, April 25, 2014 6:14:38 PM UTC-4, Paulo Correa wrote:

I`ve set up ES v.1.1.1 + AWS-cloud-plugin 2.1.1 on an EC2 instance,
using a Role to give access to my S3 bucket. Everything seems fine, I can
put, list and get objects from inside the instance (tested with node.js
code), but when I try to create an ES snapshot I get the following error
message:

$ curl -XPUT
"localhost:9200/_snapshot/my_snapshot_repo/snapshot1?wait_for_completion=true"
{"error":"SnapshotCreationException[[my_snapshot_repo:snapshot1] failed
to create snapshot]; nested: IOException[Failed to get
[snapshot-snapshot1]]; nested: AmazonClientException[Unable to unmarshall
error response (The declaration for the entity "ContentType" must end
with '>'.). Response Code: 417, Response Text: Expectation Failed]; nested:
SAXParseException[The declaration for the entity "ContentType" must end
with '>'.]; ","status":500}

Full error stack here: elasticsearch log error · GitHub

I`ve set up my snapshot repository with:

$ curl -XPUT 'http://localhost:9200/_snapshot/my_snapshot_repo' -d '{
"type": "s3", "settings": { "bucket": "mybucket","region": "sa-east"}}'

My elasticsearch.yml aws config is as follows:

...
cluster.name: montadores
cloud.aws.region: sa-east-1
discovery:
type: ec2

discovery.ec2.tag.elasticsearch_cluster: mobile_montadores
cloud.node.auto_attributes: true
...

Discovery seems OK, as I only have 1 instance running:

...

[2014-04-25 21:16:53,265][INFO ][cluster.service ] [Cap 'N Hawk]
new_master [Cap 'N
Hawk][M013xeCxQSu8QbZzdNucVA][ip-10-152-89-168][inet[/10.152.89.168:9300]]{aws_availability_zone=sa-east-1a},
reason: zen-disco-join (elected_as_master)

...

[2014-04-25 21:16:56,005][INFO ][node ] [Cap 'N Hawk]
started

The only similar thing I found to this error was an old closed issue:
Snapshotting to S3 occasionally fails · Issue #1137 · elastic/elasticsearch · GitHub

Can anybody shed some light if I`m doing something wrong?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/261e366b-232e-4474-ba47-4d01c18af569%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

I was able to resolve my issue. We have elasticsearch in a private subnet.
Access to S3 goes through squid proxy. I was able to alter the squid
proxy configuration to fix the issue.

I added the following to /etc/squid/squid.conf...

ignore_error_100 on

For reference, here is the squid proxy docs
(squid : ignore_expect_100 configuration directive)

Not sure if you are in the same situation, but hope this helps.

--Pete

On Thursday, May 8, 2014 9:51:28 AM UTC-4, Paulo Correa wrote:

Hi Pete,

we ran some more tests to see if we could narrow down the problem. We
found out that whenever we had an instance that was not in AWS' default-vpc
launch an snapshot, the problem occurred. That was the only factor that
made the problem happen, so it is not an issue with permissions, role,
buckets, etc, but rather may be a routing problem.

We're thinking of submitting a ticket to AWS Support, as we're still
facing the error when we launch an instance in our internal VPC.

Em quinta-feira, 8 de maio de 2014 10h21min48s UTC-3, Pete Michel escreveu:

Paulo,

Did you ever figure out your error? I just encountered the exact same
problem and was hoping you had found a solution

Thanks,
Pete

On Friday, April 25, 2014 6:14:38 PM UTC-4, Paulo Correa wrote:

I`ve set up ES v.1.1.1 + AWS-cloud-plugin 2.1.1 on an EC2 instance,
using a Role to give access to my S3 bucket. Everything seems fine, I can
put, list and get objects from inside the instance (tested with node.js
code), but when I try to create an ES snapshot I get the following error
message:

$ curl -XPUT
"localhost:9200/_snapshot/my_snapshot_repo/snapshot1?wait_for_completion=true"
{"error":"SnapshotCreationException[[my_snapshot_repo:snapshot1] failed
to create snapshot]; nested: IOException[Failed to get
[snapshot-snapshot1]]; nested: AmazonClientException[Unable to unmarshall
error response (The declaration for the entity "ContentType" must end
with '>'.). Response Code: 417, Response Text: Expectation Failed]; nested:
SAXParseException[The declaration for the entity "ContentType" must end
with '>'.]; ","status":500}

Full error stack here: elasticsearch log error · GitHub

I`ve set up my snapshot repository with:

$ curl -XPUT 'http://localhost:9200/_snapshot/my_snapshot_repo' -d '{
"type": "s3", "settings": { "bucket": "mybucket","region": "sa-east"}}'

My elasticsearch.yml aws config is as follows:

...
cluster.name: montadores
cloud.aws.region: sa-east-1
discovery:
type: ec2

discovery.ec2.tag.elasticsearch_cluster: mobile_montadores
cloud.node.auto_attributes: true
...

Discovery seems OK, as I only have 1 instance running:

...

[2014-04-25 21:16:53,265][INFO ][cluster.service ] [Cap 'N
Hawk] new_master [Cap 'N
Hawk][M013xeCxQSu8QbZzdNucVA][ip-10-152-89-168][inet[/10.152.89.168:9300]]{aws_availability_zone=sa-east-1a},
reason: zen-disco-join (elected_as_master)

...

[2014-04-25 21:16:56,005][INFO ][node ] [Cap 'N
Hawk] started

The only similar thing I found to this error was an old closed issue:
Snapshotting to S3 occasionally fails · Issue #1137 · elastic/elasticsearch · GitHub

Can anybody shed some light if I`m doing something wrong?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/477b6020-911b-406a-b37e-a27543fe1c7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.