Elasticsearch s3 plugin error "repository_verification_exception"

Mark_Laff · October 14, 2021, 10:01pm

I'm running ES 6.8 locally (on Mac OS X installed via Homebrew)

$ curl -sS -XGET 127.0.0.1:9200
{
  "name" : "JdRGfj8",
  "cluster_name" : "elasticsearch_brew",
  "cluster_uuid" : "PahjUQpJTpy40myvs-sKMg",
  "version" : {
    "number" : "6.8.13",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "be13c69",
    "build_date" : "2020-10-16T09:09:46.555371Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

and the S3 extension

$ /usr/local/Cellar/elasticsearch@6/6.8.13_1/bin/elasticsearch-plugin list
repository-s3

I've stored my S3 access key and secret key:

$ /usr/local/Cellar/elasticsearch@6/6.8.13_1/bin/elasticsearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

When I try to create the snapshot:

curl -sS -XPUT 127.0.0.1:9200/_snapshot/data43-es-snapshots-scratch -H 'Content-Type: application/json' -d '
 {"type": "s3",
 "settings": {
 "bucket": "data43",
 "base_path": "es-snapshots/scratch",
 "readonly": "true",
 "region": "us-gov-west-1"
 }}'

I get rejected:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[data43-es-snapshots-scratch] path [es-snapshots/scratch] is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[data43-es-snapshots-scratch] path [es-snapshots/scratch] is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Exception when listing blobs by prefix [index-]",
      "caused_by": {
        "type": "amazon_s3_exception",
        "reason": "The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 9F2J6M7Q7R4BK4XE; S3 Extended Request ID: NN/h0BoWudYpzCCclL2jxUINsHUDrhDyMbcMdmixnnI2mfn/HAqSvu/chVxljz9rdNr/hVB75nA=)"
      }
    }
  },
  "status": 500
}

Using those same ID and Secret I can access the S3 bucket just fine:

$ AWS_PROFILE=developer-qa aws s3 ls s3://data43/es-snapshots/scratch
                           PRE scratch/

So... how can I trace what request the S3 Plugin is sending to AWS? Based on the error message, it seems like the ID is not being passed correctly. Or perhaps the region is not being passed through correctly? That would also result in the ID not being found.

zx8086 · October 14, 2021, 10:45pm

@Mark_Laff

This will log the info you need to debug the issue

PUT _cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.snapshots" : "DEBUG",
    "logger.org.elasticsearch.repositories.s3": "DEBUG",
    "logger.com.amazonaws" : "DEBUG"
  }
}

You are running a very old version, upgrading might help.

warkolm · October 15, 2021, 12:40am

Welcome to our community!

Did you setup the repository, as that's not mentioned in your steps above.

Mark_Laff · October 15, 2021, 1:29am

Hi, good to be here! I'm new at some of the es terminology - I thought the "PUT /_snapshot/data43-es-snapshots-scratch" would set up the repository, right?

This query shows the created repository?

$ curl -sS -XGET http://127.0.0.1:9200/_snapshot | jq .
{
  "data43-es-snapshots-scratch": {
    "type": "s3",
    "settings": {
      "bucket": "data43",
      "base_path": "es-snapshots/scratch",
      "readonly": "true",
      "region": "us-gov-west-1"
    }
  }
}

warkolm · October 15, 2021, 1:33am

Ahh, nope
Check out Register a snapshot repository | Elasticsearch Guide [7.15] | Elastic

Mark_Laff · October 15, 2021, 1:38am

@warkolm Ah... I used the wrong terminology above - this step (above) creates the repository, no?

Mark_Laff:

curl -sS -XPUT 127.0.0.1:9200/_snapshot/data43-es-snapshots-scratch -H 'Content-Type: application/json' -d '
 {"type": "s3",
 "settings": {
 "bucket": "data43",
 "base_path": "es-snapshots/scratch",
 "readonly": "true",
 "region": "us-gov-west-1"
 }}'

warkolm · October 15, 2021, 1:39am

Yeah it does, did that work? Can you check GET /_snapshot/?

Mark_Laff · October 15, 2021, 2:26am

The GET /_snapshot returned this

$ curl -sS -XGET http://127.0.0.1:9200/_snapshot | jq .
{
  "data43-es-snapshots-scratch": {
    "type": "s3",
    "settings": {
      "bucket": "data43",
      "base_path": "es-snapshots/scratch",
      "readonly": "true",
      "region": "us-gov-west-1"
    }
  }
}

And here's the result after turning on logging -

[2021-10-14T21:44:52,376][DEBUG][o.e.r.s.S3Repository     ] [JdRGfj8] using bucket [data43], chunk_size [1gb], server_side_encryption [false], buffer_size [51.1mb], cannedACL [], storageClass []
[2021-10-14T21:44:52,448][DEBUG][c.a.s.s.AmazonS3Client   ] [JdRGfj8] Bucket region cache doesn't have an entry for data43. Trying to get bucket region from Amazon S3.
[2021-10-14T21:44:52,455][DEBUG][c.a.request              ] [JdRGfj8] Sending Request: HEAD https://data43.s3.amazonaws.com / Headers: (User-Agent: aws-sdk-java/1.11.406 Mac_OS_X/11.6 OpenJDK_64-Bit_Server_VM/17+0 java/17, amz-sdk-invocation-id: 729849c9-4684-fe7f-008d-b9f777708353, Content-Type: application/octet-stream, ) 
[2021-10-14T21:44:52,459][DEBUG][c.a.a.AWS4Signer         ] [JdRGfj8] AWS4 Canonical Request: '"HEAD
/

amz-sdk-invocation-id:729849c9-4684-fe7f-008d-b9f777708353
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:data43.s3.amazonaws.com
user-agent:aws-sdk-java/1.11.406 Mac_OS_X/11.6 OpenJDK_64-Bit_Server_VM/17+0 java/17
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20211015T014452Z

amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
[2021-10-14T21:44:52,460][DEBUG][c.a.a.AWS4Signer         ] [JdRGfj8] AWS4 String to Sign: '"AWS4-HMAC-SHA256
20211015T014452Z
20211015/us-east-1/s3/aws4_request
75e2f91dcd70c14fd181687423f71f917fe817f4db3859ed789a7b2ca2c66c4e"
[2021-10-14T21:44:52,548][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] connecting to data43.s3.amazonaws.com/52.217.172.185:443
[2021-10-14T21:44:52,549][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Connecting socket to data43.s3.amazonaws.com/52.217.172.185:443 with timeout 10000
[2021-10-14T21:44:52,588][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Enabled protocols: [TLSv1.3, TLSv1.2]
[2021-10-14T21:44:52,589][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2021-10-14T21:44:52,590][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] socket.getSupportedProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello], socket.getEnabledProtocols(): [TLSv1.3, TLSv1.2]
[2021-10-14T21:44:52,590][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1, TLSv1.3]
[2021-10-14T21:44:52,590][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Starting handshake
[2021-10-14T21:44:52,780][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Secure session established
[2021-10-14T21:44:52,780][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  negotiated protocol: TLSv1.2
[2021-10-14T21:44:52,780][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2021-10-14T21:44:52,781][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  peer principal: CN=*.s3.amazonaws.com, O="Amazon.com, Inc.", L=Seattle, ST=Washington, C=US
[2021-10-14T21:44:52,781][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  peer alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
[2021-10-14T21:44:52,781][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  issuer principal: CN=DigiCert Baltimore CA-2 G2, OU=www.digicert.com, O=DigiCert Inc, C=US
[2021-10-14T21:44:52,783][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] created: data43.s3.amazonaws.com/52.217.172.185:443
[2021-10-14T21:44:52,859][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] shutting down output of data43.s3.amazonaws.com/52.217.172.185:443
[2021-10-14T21:44:52,859][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] shutting down input of data43.s3.amazonaws.com/52.217.172.185:443
[2021-10-14T21:44:52,859][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] closing data43.s3.amazonaws.com/52.217.172.185:443
[2021-10-14T21:44:52,860][DEBUG][c.a.request              ] [JdRGfj8] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 41QGSB17A14BHK3G; S3 Extended Request ID: UkIqbHY8qZpfgiqx3MwJsQUFC81A4fi+1/FyBuJewc4hpMYJd5XF3E049ToFgH9naTa9SyQnh2w=), S3 Extended Request ID: UkIqbHY8qZpfgiqx3MwJsQUFC81A4fi+1/FyBuJewc4hpMYJd5XF3E049ToFgH9naTa9SyQnh2w=
[2021-10-14T21:44:52,860][DEBUG][c.a.s.s.AmazonS3Client   ] [JdRGfj8] Not able to derive region of the data43 from the HEAD Bucket requests.
[2021-10-14T21:44:52,860][DEBUG][c.a.s.s.AmazonS3Client   ] [JdRGfj8] Region for data43 is null
...
[there's a subsequent http request that gets rejected]

zx8086 · October 15, 2021, 6:10am

@Mark_Laff

Have you setup the correct access, how are you authenticating ?

Mark_Laff · October 15, 2021, 1:08pm

Hi zx, I stored the id and secret this way:

echo "AKIAQXI**********" | /usr/local/Cellar/elasticsearch@6/6.8.13_1/bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key
echo "q4v+R4*****************UoRnACEr"| /usr/local/Cellar/elasticsearch@6/6.8.13_1/bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key

with the actual ID and secret.

As for the IAS policy for this ID, that is set up by one of the security folks at my company. What are the key permissions required? ... And this used to work with 5.7 - it is the upgrade to 5.8 which seems to have caused the breakage.

Mark_Laff · October 18, 2021, 8:24pm

Additional information - the debug log from the S3 plugin for the PUT _snapshot/data43-es-snapshots-scratch request above shows:

[2021-10-18T16:17:51,255][DEBUG][o.e.r.s.S3Repository     ] [JdRGfj8] using bucket [data43], chunk_size [1gb], server_side_encryption [false], buffer_size [51.1mb], cannedACL [], storageClass []
[2021-10-18T16:17:51,257][DEBUG][c.a.s.s.AmazonS3Client   ] [JdRGfj8] Bucket region cache doesn't have an entry for data43. Trying to get bucket region from Amazon S3.
[2021-10-18T16:17:51,258][DEBUG][c.a.request              ] [JdRGfj8] Sending Request: HEAD https://data43.s3.amazonaws.com / Headers: (User-Agent: aws-sdk-java/1.11.406 Mac_OS_X/11.6 OpenJDK_64-Bit_Server_VM/17+0 java/17, amz-sdk-invocation-id: 0a63df19-d027-0901-7b97-59cc7925377a, Content-Type: application/octet-stream, ) 
[2021-10-18T16:17:51,259][DEBUG][c.a.a.AWS4Signer         ] [JdRGfj8] AWS4 Canonical Request: '"HEAD
/

amz-sdk-invocation-id:0a63df19-d027-0901-7b97-59cc7925377a
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:data43.s3.amazonaws.com
user-agent:aws-sdk-java/1.11.406 Mac_OS_X/11.6 OpenJDK_64-Bit_Server_VM/17+0 java/17
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20211018T201751Z

amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
[2021-10-18T16:17:51,259][DEBUG][c.a.a.AWS4Signer         ] [JdRGfj8] AWS4 String to Sign: '"AWS4-HMAC-SHA256
20211018T201751Z
20211018/us-east-1/s3/aws4_request
ae294d6b30d086b62c2aaf371c1cd0ac0cac8e7505701e5c416def9dcefdeb6c"
[2021-10-18T16:17:51,260][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] shutting down output of data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,260][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] shutting down input of data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,261][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] closing data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,261][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] connecting to data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,262][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Connecting socket to data43.s3.amazonaws.com/52.217.84.100:443 with timeout 10000
[2021-10-18T16:17:51,298][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Enabled protocols: [TLSv1.3, TLSv1.2]
[2021-10-18T16:17:51,298][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2021-10-18T16:17:51,298][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] socket.getSupportedProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello], socket.getEnabledProtocols(): [TLSv1.3, TLSv1.2]
[2021-10-18T16:17:51,299][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1, TLSv1.3]
[2021-10-18T16:17:51,299][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Starting handshake
[2021-10-18T16:17:51,483][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8] Secure session established
[2021-10-18T16:17:51,483][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  negotiated protocol: TLSv1.2
[2021-10-18T16:17:51,483][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2021-10-18T16:17:51,483][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  peer principal: CN=*.s3.amazonaws.com, O="Amazon.com, Inc.", L=Seattle, ST=Washington, C=US
[2021-10-18T16:17:51,483][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  peer alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
[2021-10-18T16:17:51,484][DEBUG][c.a.h.c.s.SdkTLSSocketFactory] [JdRGfj8]  issuer principal: CN=DigiCert Baltimore CA-2 G2, OU=www.digicert.com, O=DigiCert Inc, C=US
[2021-10-18T16:17:51,484][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] created: data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,561][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] shutting down output of data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,561][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] shutting down input of data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,562][DEBUG][c.a.i.SdkSSLSocket       ] [JdRGfj8] closing data43.s3.amazonaws.com/52.217.84.100:443
[2021-10-18T16:17:51,562][DEBUG][c.a.request              ] [JdRGfj8] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: GX699E4J5GX7YMFX; S3 Extended Request ID: 5Ajy9PDkznjT/zAYcbHe8CWEYqGeotgMiprFgd25gkG+iXeESVQQfjvxgvim1pqg3+HQnrpe1rM=), S3 Extended Request ID: 5Ajy9PDkznjT/zAYcbHe8CWEYqGeotgMiprFgd25gkG+iXeESVQQfjvxgvim1pqg3+HQnrpe1rM=
[2021-10-18T16:17:51,562][DEBUG][c.a.s.s.AmazonS3Client   ] [JdRGfj8] Not able to derive region of the data43 from the HEAD Bucket requests.

but the original request includes a bucket specification: "bucket": "data43", - shouldn't that be the name of the bucket used? In other words, should the HEAD request logged here be executed at all?

system · November 15, 2021, 8:24pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
S3 Repository plugin works on Mac/brew but not Ubuntu/apt-get Elasticsearch	4	709	August 9, 2020
Getting repository verification exception Elasticsearch	6	3899	June 28, 2020
The issue with Elasticsearch 6.8 & repository-s3 Elasticsearch snapshot-and-restore	3	1390	April 15, 2022
Cloud-aws plugin of elasticsearch 2.3.5 Elasticsearch	13	3296	December 1, 2017
Elasticsearch Create a snapshot repository error Elasticsearch	14	7483	November 4, 2022

Elasticsearch s3 plugin error "repository_verification_exception"

Related topics