Repository_verification_exception

Elastic Stack Elasticsearch
1

I am using ES 8.13.4 in the EKS cluster, On the daily snapshot we are getting snapshot exceptions like connection pool shutdown. What was the reason behind this it was working fine suddenly without any changes we are getting this error.

##Error1:

{
  "type": "repository_exception",
  "reason": "[daily-snapshot] Could not determine repository generation from root blobs",
  "caused_by": {
    "type": "i_o_exception",
    "reason": "Exception when listing blobs by prefix [index-]",
    "caused_by": {
      "type": "amazon_s3_exception",
      "reason": "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3VKADEXMMG4YS8NK; S3 Extended Request ID: 7CLwH5Smt+rWsemLKOm64+2hVVDCPgLp4g2FQT7DlGr/C9n685VKdj+6VMylt2NGwedy4=; Proxy: null)"
    }
  }
}

##Erro2

{
  "type": "snapshot_exception",
  "reason": "[daily-snapshot:vector-2024.07.15-2ktda621quq2miiyynagsq/_y97vgCARAWWUNQNfyECWQ] failed to update snapshot in repository",
  "caused_by": {
    "type": "i_o_exception",
    "reason": "Unable to upload object [meta-_y97vgCARAWWECWQ.dat] using a single upload",
    "caused_by": {
      "type": "amazon_client_exception",
      "reason": "java.lang.IllegalStateException: Connection pool shut down",
      "caused_by": {
        "type": "illegal_state_exception",
        "reason": "Connection pool shut down"
      }
    },
    "suppressed": [
      {
        "type": "i_o_exception",
        "reason": "Unable to upload object [snap-_y97vgCARAWWUNQCWQ.dat] using a single upload",
        "caused_by": {
          "type": "amazon_client_exception",
          "reason": "java.lang.IllegalStateException: Connection pool shut down",
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "Connection pool shut down"
          }
        }
      }
    ]
  }
}
2

It looks like the access key that you're using no longer has permission to access that bucket in S3.
Perhaps someone revoked the key, or changed the IAM policy.

3

We have not changed anything. earlier it was working fine, and our IAM role and policy has access to the bucket, but we have a condition like session based IAM roles, we have decoded the session also it was fine, but the the nodes not able to access it.

4

This error says otherwise.
If that error still occurs when you attempt to verify the repository then it's a very strong indication that the access is not what you think it is.
I think you're going to need to diagnose this from the the AWS/IAM side.