# Snapshot repository - Unable to refresh Azure sas\_token

**URL:** <https://discuss.elastic.co/t/snapshot-repository-unable-to-refresh-azure-sas-token/300380>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [March 22, 2022, 7:42pm UTC](https://discuss.elastic.co/t/snapshot-repository-unable-to-refresh-azure-sas-token/300380 "2022-03-22T19:42:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.elastic.co/u/mcosta)\
**Post date:** [March 22, 2022, 7:42pm UTC](https://discuss.elastic.co/t/snapshot-repository-unable-to-refresh-azure-sas-token/300380/1 "2022-03-22T19:42:56Z")

</div>

Hi all,

Our test cluster (3 nodes, version 8.1.1, on RHEL 7.9 VMs) shows "Some repositories contain errors" message.

Using the option "Verify repository" I got "Not connected" and details shows that Azure sas\_token has expired some time ago:

Expiry [Sun, **13 Feb 2022** 03:48:37 GMT] - Current [Tue, **22 Mar 2022** 19:00:29 GMT]

We didn't notice it before because this is a test cluster. QA and Prod clusters use file system repositories.

Within Azure Storage Accounts Shared Access Signature, I issued a new sas\_token, remove old token (using "Elasticsearch-keystore remove azure.client.default.sas\_token") and add the new token (using "Elasticsearch-keystore add azure.client.default.sas\_token") from every Elasticsearch node.

"Verify repository" shows same error (same expiry date! - **13 Feb 2022** ).

I then restart each Elasticsearch node and Kibana (and delete and recreate this repository from Kibana interface) but the error still the same!!

Am I missing something? How can I refresh this token?

Follows last WARN log from one Elasticsearch node.

Thank you

```auto
[2022-03-22T18:59:55,564][WARN][r.suppressed] [azlad0010.azifr01.bdso.tech] path: /_snapshot/az_backup/_all, params: {repository=az_backup, snapshot=_all}
org.elasticsearch.transport.RemoteTransportException: [azlad0008.azifr01.bdso.tech][10.237.88.15:9300][cluster:admin/snapshot/get]
Caused by: org.elasticsearch.repositories.RepositoryException: [az_backup] Could not determine repository generation from root blobs
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1888) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.1.jar:8.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: Unable to list blobs by prefix [index-] for path
        at org.elasticsearch.repositories.azure.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:339) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:135) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2591) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2563) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1885) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.1.jar:8.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: blob_storage_exception: Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:effdf741-301e-000d-2f1f-3e440d000000
Time:2022-03-22T18:59:55.5507826Z</Message><AuthenticationErrorDetail>Signature not valid in the specified time frame: Start [Fri, 12 Feb 2021 19:48:37 GMT] - Expiry [Sun, 13 Feb 2022 03:48:37 GMT] - Current [Tue, 22 Mar 2022 18:59:55 GMT]</AuthenticationErrorDetail></Error>"
        at jdk.internal.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:334) ~[?:?]
        at com.azure.core.http.rest.RestProxy.lambda$ensureExpectedStatus$5(RestProxy.java:375) ~[?:?]
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118) ~[?:?]
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) ~[?:?]
        at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:320) ~[?:?]
        at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:337) ~[?:?]
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344) ~[?:?]
        at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onSubscribe(MonoCacheTime.java:276) ~[?:?]
        at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:191) ~[?:?]
        at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) ~[?:?]
        at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[?:?]
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[?:?]
        at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:132) ~[?:?]
        at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[?:?]
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[?:?]
        at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:123) ~[?:?]
        at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:112) ~[?:?]
        at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:213) ~[?:?]
        at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:123) ~[?:?]
        at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:178) ~[?:?]
        at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96) ~[?:?]
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) ~[?:?]
        at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:121) ~[?:?]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) ~[?:?]
        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[?:?]
        at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:374) ~[?:?]
        at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:373) ~[?:?]
        at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:429) ~[?:?]
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:655) ~[?:?]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
        at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45) ~[?:?]
        at org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:27) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717) ~[elasticsearch-8.1.1.jar:8.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
        Suppressed: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[?:?]
                at reactor.core.publisher.Flux.blockLast(Flux.java:2497) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.<init>(ContinuablePagedByItemIterable.java:50) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:37) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedIterable.iterator(ContinuablePagedIterable.java:106) ~[?:?]
                at org.elasticsearch.repositories.azure.AzureBlobStore.lambda$listBlobsByPrefix$15(AzureBlobStore.java:327) ~[?:?]
                at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46) ~[?:?]
                at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
                at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45) ~[?:?]
                at org.elasticsearch.repositories.azure.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:321) ~[?:?]
                at org.elasticsearch.repositories.azure.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:135) ~[?:?]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2591) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2563) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1885) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.1.jar:8.1.1]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
                at java.lang.Thread.run(Thread.java:833) [?:?]

```

---

<div class="post-metadata">

**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.elastic.co/u/mcosta)\
**Post date:** [March 23, 2022, 6:19pm UTC](https://discuss.elastic.co/t/snapshot-repository-unable-to-refresh-azure-sas-token/300380/2 "2022-03-23T18:19:10Z")

</div>

SOLVED!

Just in case someone get same error:

Executing a reload solve the problem 😀

POST \_nodes/reload\_secure\_settings  
{  
"secure\_settings\_password": ""  
}

Restarting nodes don't reload keystore values... 🤔

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 20, 2022, 6:19pm UTC](https://discuss.elastic.co/t/snapshot-repository-unable-to-refresh-azure-sas-token/300380/3 "2022-04-20T18:19:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.