Hi,
Elasticsearch 8.2.3
I'm trying to add an S3 snapshot repository, the endpoint is Yandexcloud S3
I've set the keys in the keystore on all master nodes, check it with show
elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key
Then I add the snapshot repository:
PUT _snapshot/ycs3
{
"type": "s3",
"settings": {
"bucket": "es-test",
"endpoint": "storage.yandexcloud.net",
"client": "default",
"base_path": "path",
"canned_acl": "bucket-owner-full-control"
}
}
Which returnes error:
{
"error" : {
"root_cause" : [
{
"type" : "repository_verification_exception",
"reason" : "[ycs3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
"suppressed" : [
{
"type" : "repository_verification_exception",
"reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]"
},
{
"type" : "repository_verification_exception",
"reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]"
}
]
}
],
"type" : "repository_verification_exception",
"reason" : "[ycs3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
"suppressed" : [
{
"type" : "repository_verification_exception",
"reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [path/tests-UY6djZJTQse3B1gdOrwmpA/data-PU4UCt3dR5S6_r8LuMhNLA.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Network is unreachable"
}
}
}
},
{
"type" : "repository_verification_exception",
"reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [path/tests-UY6djZJTQse3B1gdOrwmpA/data-QLd6TNjhQHWKNgLpm3PzFg.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Network is unreachable"
}
}
}
}
]
},
"status" : 500
}
I see a folder with files created by the ES in the bucket
path/tests-UY6djZJTQse3B1gdOrwmpA
Then i take snapshot, got error:
INTERNAL_SERVER_ERROR: IndexShardSnapshotFailedException[Failed to write shard level snapshot metadata for [snapshot-name-4bb7krxwtk6hp_v3ovsz5a/ROb80jvRR0WH2DNmdbUnfg] to [index-_14V5serSM6Ub6UM79c1Nw]]; nested: IOException[Unable to upload object [path/indices/HEDN0S9lTm-iQZm9bKGxiQ/0/index-_14V5serSM6Ub6UM79c1Nw] using a single upload]; nested: SdkClientException[Failed to connect to service endpoint: ]; nested: SocketException[Network is unreachable]
path/indexes/HEDN0S9lTm-iQZm9bKGxiQ/meta-.dat file appears in the bucket
S3 ACL is fine, i can read write to a bucket