Amazon_s3_exception when trying to create repo, even though it still creates test folder

[javadevmtl (None)]

Hi getting the below error, but the funny thing is there's a test folder created in the bucker. For example: tests--v15dKjeS5uBICtqKNVEdg/ and the master.dat file is inside.

{
  "type": "s3",
  "settings": {
    "bucket": "xxxxxx",
    "client": "xxxxxx"
  }
}

This error still create some folders in the bucket.

{
    "error": {
        "root_cause": [
            {
                "type": "amazon_s3_exception",
                "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxx; S3 Extended Request ID: xxxxxx)"
            }
        ],
        "type": "amazon_s3_exception",
        "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxx; S3 Extended Request ID: xxxxxx)"
    },
    "status": 500
}

Furthermore I get other random errors running the same request above. When this happens no folder is created.

{
    "error": {
        "root_cause": [
            {
                "type": "repository_verification_exception",
                "reason": "[xxxxxx] path  is not accessible on master node"
            }
        ],
        "type": "repository_verification_exception",
        "reason": "[xxxxxx] path  is not accessible on master node",
        "caused_by": {
            "type": "i_o_exception",
            "reason": "Unable to upload object [tests-GBYZhaObSEaz_O6Nf2Zjww/master.dat] using a single upload",
            "caused_by": {
                "type": "amazon_s3_exception",
                "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxx; S3 Extended Request ID: xxxxxx)"
            }
        }
    },
    "status": 500
}

[stephenb (Stephen Brown)]

Just a shot in the dark....

Curious is this the only Cluster with access with this Repo? Are you trying to restore from another cluster?

1. If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. On other clusters, register the repository as read-only.

2. This prevents multiple clusters from writing to the repository at the same time and corrupting the repository’s contents. It also prevents Elasticsearch from caching the repository’s contents, which means that changes made by other clusters will become visible straight away.

[javadevmtl (None)]

No. It's single cluster and I'm trying to create a new repo.

I have 2 repos with 2 separate buckets, 1 running under "default" client and this new one.

The default one started failing with 403 forbidden errors as well and for the life of me can'y seem to figure out why...

So I tried to create a new repo to test. I created separate key/secret entries in the keystore and even used POST _nodes/reload_secure_settings

The old repo fails with...

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[s3-repository] [[p3QwHknxSfesEYFFwEAfJw, 'RemoteTransportException[[data-0004][xxx.xxx.xxx.59:9300][internal:admin/repository/verify]]; nested: BlobStoreException[Failed to check if blob [master.dat] exists]; nested: NotSerializableExceptionWrapper[amazon_s3_exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: XXXXXX; S3 Extended Request ID: XXXXXX)];'], [MNNLmMoVRDS2HZWVdnN4-A, 'RemoteTransportException[[data-0001][xxx.xxx.xxx.20:9300][internal:admin/repository/verify]]; nested: BlobStoreException[Failed to check if blob [master.dat] exists]; nested: NotSerializableExceptionWrapper[amazon_s3_exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: XXXXXX; S3 Extended Request ID: XXXXXX)];']]"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[s3-repository] [[p3QwHknxSfesEYFFwEAfJw, 'RemoteTransportException[[data-0004][xxx.xxx.xxx.59:9300][internal:admin/repository/verify]]; nested: BlobStoreException[Failed to check if blob [master.dat] exists]; nested: NotSerializableExceptionWrapper[amazon_s3_exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: XXXXXX; S3 Extended Request ID: XXXXXX)];'], [MNNLmMoVRDS2HZWVdnN4-A, 'RemoteTransportException[[data-0001][xxx.xxx.xxx.20:9300][internal:admin/repository/verify]]; nested: BlobStoreException[Failed to check if blob [master.dat] exists]; nested: NotSerializableExceptionWrapper[amazon_s3_exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: XXXXXX; S3 Extended Request ID: XXXXXX)];']]"
  },
  "status": 500
}