# Unknown issue with Elastic snapshots **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [July 12, 2021, 4:04pm UTC](https://discuss.elastic.co/t/unknown-issue-with-elastic-snapshots/278442 "2021-07-12T16:04:58Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![kcarsey](https://avatars.discourse-cdn.com/v4/letter/k/97f17d/32.png) [@kcarsey](https://discuss.elastic.co/u/kcarsey)\ **Post date:** [July 12, 2021, 4:04pm UTC](https://discuss.elastic.co/t/unknown-issue-with-elastic-snapshots/278442/1 "2021-07-12T16:04:58Z")
I will provide a bit of backstory information here as I think it could be relevant. I run several ec2 Elastic clusters in AWS, all being backed up by snapshot to aws s3. I have recently created two new clusters in us-east-2. The snap s3 buckets were origionally created in the wrong region by accident. I attempted to add a snap repo to cluster1 and did not realize the bucket was in the wrong region until I was getting a transport error (we use s3 endpoint inside vpc). I opened an internal ticket and had buckets deleted and re-created, approx 3 days ago. Now I am getting a very strange error that I have never seen from cluster1. ```auto { "error": { "root_cause": [ { "type": "repository_verification_exception", "reason": "[qm-na6] path is not accessible on master node" } ], "type": "repository_verification_exception", "reason": "[qm-na6] path is not accessible on master node", "caused_by": { "type": "i_o_exception", "reason": "Unable to upload object [tests-vxHMJk-aTeGhpFXN_PREEQ/master.dat] using a single upload", "caused_by": { "type": "amazon_s3_exception", "reason": "amazon_s3_exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 1XS6JH52NW4BWK6T; S3 Extended Request ID: ZEh5U6D7U0kUd5GGiWnngLU5xmcbIAwHOF7wWSaYxLJSNJ8/w5n7K19bAlykLAXmeFdeE8VYlfw=)" } } }, "status": 500 } ``` All of my ec2/elastic cluster setup is controlled by terraform/ansible, they are all using consistant configuration/IAM Role to allow permission to s3 bucket. In fact, Cluster2 is using the exact same IAM role and works without issue. ```auto { "Statement": [ { "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads", "s3:ListBucketVersions" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket2" ] }, { "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*" ] } ], "Version": "2012-10-17" } ``` It seems as if some setting, or something may have been "cached" somewhere in elastic config? I use Kibana to create the snapshot repo and validate, with no luck. I have also tried specifying the region manually in the snap repo, which normally I do not have to do, but still no luck. Here is error when manually specifying region: ```auto { "error": { "root_cause": [ { "type": "repository_verification_exception", "reason": "[qm-na6] [[91KomCdaRN63dj6PItTSLg, 'RemoteTransportException[[REDACTED.net][10.125.52.89:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];'], [QO5rq9A0SqGeOCaLgiqtbA, 'RemoteTransportException[[REDACTED.net][10.125.52.96:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];'], [ow39ZlVLQyaryKluekPBGA, 'RemoteTransportException[[REDACTED.net][10.125.48.159:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];'], [GdeKk7HTSSKQ3GpSvWPTig, 'RemoteTransportException[[REDACTED.net][10.125.48.250:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];']]" } ], "type": "repository_verification_exception", "reason": "[qm-na6] [[91KomCdaRN63dj6PItTSLg, 'RemoteTransportException[[REDACTED.net][10.125.52.89:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];'], [QO5rq9A0SqGeOCaLgiqtbA, 'RemoteTransportException[[REDACTED.net][10.125.52.96:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];'], [ow39ZlVLQyaryKluekPBGA, 'RemoteTransportException[[REDACTED.net][10.125.48.159:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];'], [GdeKk7HTSSKQ3GpSvWPTig, 'RemoteTransportException[[REDACTED.net][10.125.48.250:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[qm-na6] missing];']]" }, "status": 500 } ``` Here is snap repo with Region added. ```auto { "type": "s3", "settings": { "bucket": "qm-elastic-snapshot-na6", "canned_acl": "bucket-owner-full-control", "storage_class": "standard_ia", "region": "us-east-2" } } ``` ---
**Author:** ![kcarsey](https://avatars.discourse-cdn.com/v4/letter/k/97f17d/32.png) [@kcarsey](https://discuss.elastic.co/u/kcarsey)\ **Post date:** [July 18, 2021, 1:49pm UTC](https://discuss.elastic.co/t/unknown-issue-with-elastic-snapshots/278442/2 "2021-07-18T13:49:27Z")
I have found the issue. After a full cluster restart I was getting the same error, RemoteTransportException. I did some more searching and it lead me to check plugins. I found that part of my nodes were missing the repository-s3 plugin. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 15, 2021, 1:50pm UTC](https://discuss.elastic.co/t/unknown-issue-with-elastic-snapshots/278442/3 "2021-08-15T13:50:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.