Snapshot repository - Unable to refresh Azure sas_token

Hi all,

Our test cluster (3 nodes, version 8.1.1, on RHEL 7.9 VMs) shows "Some repositories contain errors" message.

Using the option "Verify repository" I got "Not connected" and details shows that Azure sas_token has expired some time ago:

Expiry [Sun, 13 Feb 2022 03:48:37 GMT] - Current [Tue, 22 Mar 2022 19:00:29 GMT]

We didn't notice it before because this is a test cluster. QA and Prod clusters use file system repositories.

Within Azure Storage Accounts Shared Access Signature, I issued a new sas_token, remove old token (using "Elasticsearch-keystore remove azure.client.default.sas_token") and add the new token (using "Elasticsearch-keystore add azure.client.default.sas_token") from every Elasticsearch node.

"Verify repository" shows same error (same expiry date! - 13 Feb 2022).

I then restart each Elasticsearch node and Kibana (and delete and recreate this repository from Kibana interface) but the error still the same!!

Am I missing something? How can I refresh this token?

Follows last WARN log from one Elasticsearch node.

Thank you

[2022-03-22T18:59:55,564][WARN ][r.suppressed             ] [azlad0010.azifr01.bdso.tech] path: /_snapshot/az_backup/_all, params: {repository=az_backup, snapshot=_all}
org.elasticsearch.transport.RemoteTransportException: [azlad0008.azifr01.bdso.tech][10.237.88.15:9300][cluster:admin/snapshot/get]
Caused by: org.elasticsearch.repositories.RepositoryException: [az_backup] Could not determine repository generation from root blobs
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1888) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.1.jar:8.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: Unable to list blobs by prefix [index-] for path
        at org.elasticsearch.repositories.azure.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:339) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:135) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2591) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2563) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1885) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.1.jar:8.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: blob_storage_exception: Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:effdf741-301e-000d-2f1f-3e440d000000
Time:2022-03-22T18:59:55.5507826Z</Message><AuthenticationErrorDetail>Signature not valid in the specified time frame: Start [Fri, 12 Feb 2021 19:48:37 GMT] - Expiry [Sun, 13 Feb 2022 03:48:37 GMT] - Current [Tue, 22 Mar 2022 18:59:55 GMT]</AuthenticationErrorDetail></Error>"
        at jdk.internal.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:334) ~[?:?]
        at com.azure.core.http.rest.RestProxy.lambda$ensureExpectedStatus$5(RestProxy.java:375) ~[?:?]
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118) ~[?:?]
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) ~[?:?]
        at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:320) ~[?:?]
        at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:337) ~[?:?]
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344) ~[?:?]
        at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onSubscribe(MonoCacheTime.java:276) ~[?:?]
        at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:191) ~[?:?]
        at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) ~[?:?]
        at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[?:?]
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[?:?]
        at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:132) ~[?:?]
        at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[?:?]
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[?:?]
        at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:123) ~[?:?]
        at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:112) ~[?:?]
        at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:213) ~[?:?]
        at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:123) ~[?:?]
        at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:178) ~[?:?]
        at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96) ~[?:?]
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) ~[?:?]
        at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:121) ~[?:?]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) ~[?:?]
        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[?:?]
        at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:374) ~[?:?]
        at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:373) ~[?:?]
        at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:429) ~[?:?]
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:655) ~[?:?]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
        at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45) ~[?:?]
        at org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:27) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717) ~[elasticsearch-8.1.1.jar:8.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
        Suppressed: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[?:?]
                at reactor.core.publisher.Flux.blockLast(Flux.java:2497) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.<init>(ContinuablePagedByItemIterable.java:50) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:37) ~[?:?]
                at com.azure.core.util.paging.ContinuablePagedIterable.iterator(ContinuablePagedIterable.java:106) ~[?:?]
                at org.elasticsearch.repositories.azure.AzureBlobStore.lambda$listBlobsByPrefix$15(AzureBlobStore.java:327) ~[?:?]
                at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46) ~[?:?]
                at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
                at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45) ~[?:?]
                at org.elasticsearch.repositories.azure.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:321) ~[?:?]
                at org.elasticsearch.repositories.azure.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:135) ~[?:?]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2591) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2563) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1885) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) ~[elasticsearch-8.1.1.jar:8.1.1]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.1.jar:8.1.1]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
                at java.lang.Thread.run(Thread.java:833) [?:?]

SOLVED!

Just in case someone get same error:

Executing a reload solve the problem :grinning:

POST _nodes/reload_secure_settings
{
"secure_settings_password": ""
}

Restarting nodes don't reload keystore values... :thinking:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.