# Repository\_verification\_exception while creating new repository in snapshot

**URL:** <https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [February 17, 2022, 11:33am UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483 "2022-02-17T11:33:36Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 17, 2022, 11:33am UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/1 "2022-02-17T11:33:36Z")

</div>

we are running 3 node Elasticsearch setup where all of the nodes are master in kubernetes .  
i have installed s3 plugin for Elasticsearch and provided access key and secret key .  
whenever i create a repository and verify the status it sends the test data to s3 bucket which is visible in my aws s3 bucket but i get this error while verifying the status :

```auto
[new] [[1GNRcQ60Qc2NbM6dv2seaQ, 'RemoteTransportException[[elasticsearch-master-2][10.2.4.31:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[new] missing];'], [u1JYwJTlSn-jYg_glq5g1A, 'RemoteTransportException[[elasticsearch-master-0][10.2.5.16:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[new] missing];']]

```

i am unable to get answer to this error in any of the threads !

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 17, 2022, 12:00pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/2 "2022-02-17T12:00:15Z")

</div>

What version are you running?

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 17, 2022, 12:03pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/3 "2022-02-17T12:03:29Z")

</div>

we are using 7.11.2

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 17, 2022, 12:12pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/4 "2022-02-17T12:12:29Z")

</div>

Ok I think there will be messages in the logs on `elasticsearch-master-0` and `elasticsearch-master-2` describing the problem.

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 17, 2022, 12:30pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/5 "2022-02-17T12:30:20Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/2/5/255db50b20ea70f786b26ad3bb17b2c9bc36f539.png)  
ELASTICSEARCH\_MASTER\_0

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/2/f237acaa20743f16ba02c67298609a04dab7f63c.png)  
ELASTICSEARCH\_MASTER\_1

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a087355424c36851d9c720f1d8c2d14de81b8e84.png)  
ELASTICSEARCH\_MASTER\_2

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 17, 2022, 12:42pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/6 "2022-02-17T12:42:32Z")

</div>

Please don't share screenshots of logs, they're impossible for other people to search (or even read sometimes).

This isn't the only message in those logs. There will be other messages that are also important.

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 17, 2022, 12:44pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/7 "2022-02-17T12:44:36Z")

</div>

so should i send full log here ?

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 17, 2022, 12:50pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/8 "2022-02-17T12:50:21Z")

</div>

```auto
{"type": "server", "timestamp": "2022-02-17T12:17:50,472Z", "level": "WARN", "component": "o.e.r.VerifyNodeRepositoryAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "[test] failed to verify repository", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "1GNRcQ60Qc2NbM6dv2seaQ" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryMissingException: [test] missing",
"at org.elasticsearch.repositories.RepositoriesService.repository(RepositoriesService.java:398) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:116) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$400(VerifyNodeRepositoryAction.java:38) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:149) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:144) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:213) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:307) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:418) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:220) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:328) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:389) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:400) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:325) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:266) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:178) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:314) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:294) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

```

Elasticsearch-master-0

```auto
{"type": "server", "timestamp": "2022-02-17T12:17:50,471Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "path: /_snapshot/test/_verify, params: {repository=test}", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "Ax82gyRaSLivrFjS2wJryA" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryVerificationException: [test] [[1GNRcQ60Qc2NbM6dv2seaQ, 'RemoteTransportException[[elasticsearch-master-2][10.2.4.31:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[test] missing];'], [u1JYwJTlSn-jYg_glq5g1A, 'RemoteTransportException[[elasticsearch-master-0][10.2.5.16:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[test] missing];']]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.finishVerification(VerifyNodeRepositoryAction.java:109) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$000(VerifyNodeRepositoryAction.java:38) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$1.handleException(VerifyNodeRepositoryAction.java:98) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1288) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1288) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:266) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:213) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:264) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:256) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:120) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:689) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:131) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:106) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:71) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:63) [transport-netty4-client-7.11.2.jar:7.11.2]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

```

ELASTICSEARCH\_MASTER\_1

```auto
{"type": "server", "timestamp": "2022-02-17T12:17:50,472Z", "level": "WARN", "component": "o.e.r.VerifyNodeRepositoryAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "[test] failed to verify repository", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "1GNRcQ60Qc2NbM6dv2seaQ" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryMissingException: [test] missing",
"at org.elasticsearch.repositories.RepositoriesService.repository(RepositoriesService.java:398) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:116) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$400(VerifyNodeRepositoryAction.java:38) ~[elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:149) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:144) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:213) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:307) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:418) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:220) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:328) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:389) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:400) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:325) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:266) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:178) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:314) [x-pack-security-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:294) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.11.2.jar:7.11.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.11.2.jar:7.11.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

```

ELASTICSEARCH\_MASTER\_2

i am sending this logs which i got whenever i hit the verify repository button

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 17, 2022, 2:16pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/9 "2022-02-17T14:16:51Z")

</div>

This is still just the one message in each log tho? You'll need to look at earlier messages too, maybe going back all the way to when the nodes last started up.

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 17, 2022, 2:25pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/10 "2022-02-17T14:25:04Z")

</div>

i am putting whole log files of all Elasticsearch nodes here as i cant post whole log in this thread .  
[master-0](https://justpaste.it/1vmo4)  
[master-1](https://justpaste.it/8lrpr)  
[master-2](https://justpaste.it/5dvmp)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 17, 2022, 2:42pm UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/11 "2022-02-17T14:42:41Z")

</div>

Thanks. The problem is here:

```auto
{"type": "server", "timestamp": "2022-02-17T10:46:05,364Z", "level": "WARN", "component": "o.e.r.RepositoriesService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to create repository [new]", "cluster.uuid": "IuPHrM-aRJSSzAM9Narm_g", "node.id": "u1JYwJTlSn-jYg_glq5g1A" , 
"stacktrace": ["org.elasticsearch.repositories.RepositoryException: [new] repository type [s3] does not exist",

```

You haven't installed the plugin on every node.

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 18, 2022, 5:37am UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/12 "2022-02-18T05:37:50Z")

</div>

okay , thanks for help @DavidTurner  
just one more thing i want to ask that how can i install s3 plugin in all nodes and made it persist even after the pod restarts or deletes as we are running Elasticsearch on kubernetes !

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 18, 2022, 8:18am UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/13 "2022-02-18T08:18:08Z")

</div>

Not sure about how to arrange that on Kubernetes sorry.

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 18, 2022, 9:35am UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/14 "2022-02-18T09:35:33Z")

</div>

okay , i have manually installed s3 plugin in all 3 nodes and manually added accesskey and secret key in keystore.  
i am getting this error while verifying the repository.

```auto
{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[letstestit] path is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[letstestit] path is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-6Ga-7hSNSQmeucUnuY9B9g/master.dat] using a single upload",
      "caused_by": {
        "type": "amazon_s3_exception",
        "reason": "amazon_s3_exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: J4J67FVP5RVNFENA; S3 Extended Request ID: OENp0t5KUdhVK3pZJsU4hIR5PsVTczYeNkbOCJ5Bosw6CkXcf4SxnP9yB7al3ewv0rd3vSD7Vxc=)"
      }
    }
  },
  "status": 500
}

```

i am attaching the log file along with it of one node so you can get more info about this .  
[https://justpaste.it/1vmo4](https://justpaste.it/1vmo4)

i have manually tested that accesskey and private key

---

<div class="post-metadata">

**Author:** ![Darshil\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshil_shah/32/99145_2.png) [@Darshil\_Shah](https://discuss.elastic.co/u/Darshil_Shah)\
**Post date:** [February 18, 2022, 10:55am UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/15 "2022-02-18T10:55:35Z")

</div>

Thanks @DavidTurner i figured out

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 18, 2022, 10:56am UTC](https://discuss.elastic.co/t/repository-verification-exception-while-creating-new-repository-in-snapshot/297483/16 "2022-03-18T10:56:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.