I have a custom s3 storage which I can communicate with using AWS sdk. I have configured the repository with Elasticsearch v7.17 and it is working fine, but after upgrade to the version v8.11 the verification to the repository failed.
I have the very same configuration between these two versions, the only thing that changed is the number of nodes, now I've scale out for 7 nodes (were 4 initially)
{
"error" : {
"root_cause" : [
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] [[1jCY0j1kR6CQOj1gQPkrhQ, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [s-4qqHILSz6SGaHq0jCp2g, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [Z2n2zAZjQauZbautWB0Lhg, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [TRLaEAzbTkKqX1bjqgK5Ww, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [dkGE4Au0ScqMnhMBtLw2Jw, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [rbtZa2WLTFylokPYQLQi7w, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]']]",
"suppressed" : [
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [] is not accessible on the node [{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0}]"
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [] is not accessible on the node [{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}]"
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [] is not accessible on the node [{8.11.1}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}]"
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [] is not accessible on the node [{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0}]"
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [] is not accessible on the node [{8.11.1}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}]"
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [] is not accessible on the node [{8.11.1}{7000099-8500003}{xpack.installed=true, ml.config_version=11.0.0, transform.config_version=10.0.0}]"
}
]
}
],
"type" : "repository_verification_exception",
"reason" : "[s3-repo] [[1jCY0j1kR6CQOj1gQPkrhQ, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [s-4qqHILSz6SGaHq0jCp2g, 'org.elasticsearch.transport.RemoteTransportException:[internal:admin/repository/verify]'], [Z2n2zAZjQauZbautWB0Lhg, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [TRLaEAzbTkKqX1bjqgK5Ww, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [dkGE4Au0ScqMnhMBtLw2Jw, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]'], [rbtZa2WLTFylokPYQLQi7w, 'org.elasticsearch.transport.RemoteTransportException: [internal:admin/repository/verify]']]",
"suppressed" : [
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [] is not accessible on the node [{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [tests--x7DQun6QrCCFPnTTurfcg/data-1jCY0j1kR6CQOj1gQPkrhQ.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [elastictst] is not accessible on the node [{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [tests--x7DQun6QrCCFPnTTurfcg/data-s-4qqHILSz6SGaHq0jCp2g.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [elastictst] is not accessible on the node [{8.11.1}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [tests--x7DQun6QrCCFPnTTurfcg/data-Z2n2zAZjQauZbautWB0Lhg.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [elastictst] is not accessible on the node [{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [tests--x7DQun6QrCCFPnTTurfcg/data-TRLaEAzbTkKqX1bjqgK5Ww.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [elastictst] is not accessible on the node [{8.11.1}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [tests--x7DQun6QrCCFPnTTurfcg/data-dkGE4Au0ScqMnhMBtLw2Jw.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
},
{
"type" : "repository_verification_exception",
"reason" : "[s3-repo] store location [elastictst] is not accessible on the node [{8.11.1}{7000099-8500003}{xpack.installed=true, ml.config_version=11.0.0, transform.config_version=10.0.0}]",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [tests--x7DQun6QrCCFPnTTurfcg/data-rbtZa2WLTFylokPYQLQi7w.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
}
]
},
"status" : 500
}
I've also got this exception on master node.
doPrivileged(AccessController.java:571)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:132)\n\t\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:119)\n\t\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3274)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:139)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:171)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\t\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:565)\n\t\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:618)\n\t\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:607)\n\t\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:668)\n\t\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:310)\n\t\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:113)\n\t\tat org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)\n\t\tat org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\n\t\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94)\n\t\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:261)\n\t\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199)\n\t\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.authenticate(ServerTransportFilter.java:126)\n\t\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:104)\n\t\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:629)\n\t\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)\n\t\tat org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:288)\n\t\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:301)\n\t\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\t\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\t\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\t\tat java.lang.Thread.run(Thread.java:1583)\n\tCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80)\n\t\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\t\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:319)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\t\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:299)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\t\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:434)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:319)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:434)\n\t\t... 32 more\n\tCaused by: java.io.IOException: Connect timed out\n\t\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\t\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)\n\t\tat java.net.Socket.connect(Socket.java:751)\n\t\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:531)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:636)\n\t\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:280)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:386)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:408)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1304)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1282)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1052)\n\t\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:95)\n\t\tat com.amazon
Feb 15 11:13:22 ch-sv04197 es01[198728]: aws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t\t... 64 more\n"}
Someone can help me with it please?