Repository_verification_exception while creating new repository in snapshot

Elastic Stack Elasticsearch
1

we are running 3 node Elasticsearch setup where all of the nodes are master in kubernetes .
i have installed s3 plugin for Elasticsearch and provided access key and secret key .
whenever i create a repository and verify the status it sends the test data to s3 bucket which is visible in my aws s3 bucket but i get this error while verifying the status :

[new] [[1GNRcQ60Qc2NbM6dv2seaQ, 'RemoteTransportException[[elasticsearch-master-2][10.2.4.31:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[new] missing];'], [u1JYwJTlSn-jYg_glq5g1A, 'RemoteTransportException[[elasticsearch-master-0][10.2.5.16:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[new] missing];']]

i am unable to get answer to this error in any of the threads !

2

What version are you running?

3

we are using 7.11.2

4

Ok I think there will be messages in the logs on elasticsearch-master-0 and elasticsearch-master-2 describing the problem.

5

image
image1366×768 179 KB

ELASTICSEARCH_MASTER_0

image
image1366×768 166 KB

ELASTICSEARCH_MASTER_1

image
image1366×768 179 KB

ELASTICSEARCH_MASTER_2

6

Please don't share screenshots of logs, they're impossible for other people to search (or even read sometimes).

This isn't the only message in those logs. There will be other messages that are also important.

7

so should i send full log here ?

8 
{"type": "server", "timestamp": "2022-02-17T12:17:50,472Z", "level": "WARN", "component": "o.e.r.VerifyNodeRepositoryAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "[test] failed to verify repository", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "1GNRcQ60Qc2NbM6dv2seaQ" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryMissingException: [test] missing",
"at org.elasticsearch.repositories.RepositoriesService.repository(RepositoriesService.java:398) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:116) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$400(VerifyNodeRepositoryAction.java:38) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:149) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:144) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:213) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:307) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:418) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:220) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:328) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:389) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:400) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:325) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:266) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:178) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:314) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:294) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

Elasticsearch-master-0

{"type": "server", "timestamp": "2022-02-17T12:17:50,471Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "path: /_snapshot/test/_verify, params: {repository=test}", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "Ax82gyRaSLivrFjS2wJryA" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryVerificationException: [test] [[1GNRcQ60Qc2NbM6dv2seaQ, 'RemoteTransportException[[elasticsearch-master-2][10.2.4.31:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[test] missing];'], [u1JYwJTlSn-jYg_glq5g1A, 'RemoteTransportException[[elasticsearch-master-0][10.2.5.16:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[test] missing];']]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.finishVerification(VerifyNodeRepositoryAction.java:109) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$000(VerifyNodeRepositoryAction.java:38) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$1.handleException(VerifyNodeRepositoryAction.java:98) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1288) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1288) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:266) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:213) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:264) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:256) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:120) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:689) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:131) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:106) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:71) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:63) [transport-netty4-client-7.11.2.jar:7.11.2]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

ELASTICSEARCH_MASTER_1

{"type": "server", "timestamp": "2022-02-17T12:17:50,472Z", "level": "WARN", "component": "o.e.r.VerifyNodeRepositoryAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "[test] failed to verify repository", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "1GNRcQ60Qc2NbM6dv2seaQ" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryMissingException: [test] missing",
"at org.elasticsearch.repositories.RepositoriesService.repository(RepositoriesService.java:398) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:116) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$400(VerifyNodeRepositoryAction.java:38) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:149) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:144) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:213) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:307) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:418) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:220) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:328) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:389) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:400) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:325) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:266) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:178) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:314) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:294) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

ELASTICSEARCH_MASTER_2

i am sending this logs which i got whenever i hit the verify repository button

9

This is still just the one message in each log tho? You'll need to look at earlier messages too, maybe going back all the way to when the nodes last started up.

10

i am putting whole log files of all Elasticsearch nodes here as i cant post whole log in this thread .
master-0
master-1
master-2

11

Thanks. The problem is here:

{"type": "server", "timestamp": "2022-02-17T10:46:05,364Z", "level": "WARN", "component": "o.e.r.RepositoriesService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to create repository [new]", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "u1JYwJTlSn-jYg_glq5g1A" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryException: [new] repository type [s3] does not exist",

You haven't installed the plugin on every node.

12

okay , thanks for help @DavidTurner
just one more thing i want to ask that how can i install s3 plugin in all nodes and made it persist even after the pod restarts or deletes as we are running Elasticsearch on kubernetes !

13

Not sure about how to arrange that on Kubernetes sorry.

14

okay , i have manually installed s3 plugin in all 3 nodes and manually added accesskey and secret key in keystore.
i am getting this error while verifying the repository.

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[letstestit] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[letstestit] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-6Ga-7hSNSQmeucUnuY9B9g/master.dat] using a single upload",
      "caused_by": {
        "type": "amazon_s3_exception",
        "reason": "amazon_s3_exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: J4J67FVP5RVNFENA; S3 Extended Request ID: OENp0t5KUdhVK3pZJsU4hIR5PsVTczYeNkbOCJ5Bosw6CkXcf4SxnP9yB7al3ewv0rd3vSD7Vxc=)"
      }
    }
  },
  "status": 500
}

i am attaching the log file along with it of one node so you can get more info about this .
https://justpaste.it/1vmo4

i have manually tested that accesskey and private key

15

Thanks @DavidTurner i figured out

1 Like
16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.