Elasticsearch snapshots fail everyday

Elastic Stack Elasticsearch
,
1

Hi,

I am using elasticsearch 8.7 in out production cluster on Azure kubernetes platform which has 8 data and master nodes and almost 4TB per node disks being used to store our observability data. we have 699 indexes as of now and almost same number of primary and replica shards.

we are using azure blob storage for snapshots and they were being successfully done till last week since when below strange errors are causing them to fail everyday. can anybody help to solve it please?

INTERNAL_SERVER_ERROR: UncategorizedExecutionException[Failed execution]; nested: ExecutionException[java.io.IOException: Unable to write blob indices/cQV-28f_Q4G_xbuxagrf9g/0/__pr8s4d-PR2Cf6o_odhWGrA]; nested: IOException[Unable to write blob indices/cQV-28f_Q4G_xbuxagrf9g/0/__pr8s4d-PR2Cf6o_odhWGrA]; nested: AbortedSnapshotException[aborted]
2

Is there more to the Elasticsearch logs you can share?

3

Hi,

Thanks for your response. i am sharing logs in see related to snapshots

master node:

{"@timestamp":"2023-06-13T01:30:00.001Z", "log.level": "INFO", "message":"starting SLM retention snapshot cleanup task", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:00.001Z", "log.level": "INFO", "message":"snapshot lifecycle policy [daily-snapshot-everything] issuing create snapshot [daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotLifecycleTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:00.001Z", "log.level": "INFO", "message":"snapshot lifecycle policy job [daily-snapshot-everything-7] issued new snapshot creation for [daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa] successfully", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotLifecycleTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:00.274Z", "log.level": "INFO", "message":"SLM retention snapshot cleanup task complete", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][snapshot_meta][T#1896]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:03.854Z", "log.level": "INFO", "message":"snapshot [snapshot-repo-azure:daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa/0Av1n922RS-6er5nVpqS_g] started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.snapshots.SnapshotsService","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:57:40.930Z", "log.level":"ERROR", "message":"failed to create snapshot for snapshot lifecycle policy [daily-snapshot-everything]: org.elasticsearch.snapshots.SnapshotException: [snapshot-repo-azure:daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa] failed to create snapshot successfully, 18 out of 740 total shards failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][snapshot][T#475]","log.logger":"org.elasticsearch.xpack.slm.SnapshotLifecycleTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:57:40.930Z", "log.level": "INFO", "message":"snapshot [snapshot-repo-azure:daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa/0Av1n922RS-6er5nVpqS_g] completed with state [PARTIAL]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][snapshot][T#475]","log.logger":"org.elasticsearch.snapshots.SnapshotsService","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}

I also noticed that when i trigger snapshot...it initiates a cluster event for obvious reasons and when any of node does not process it within 10 seconds then master kicks that node out and it rejoins after some time. i suspect snapshot fails for this reason but i dont know how to fix kit.

i have 8 nodes with 24GB memory limits and 12GB heap size.

4

Logs from Data node:

{"@timestamp":"2023-06-13T09:10:57.569Z", "log.level": "WARN", "message":"[[qa2-cust-eastus-logs-2023.06.13][0]][snapshot-repo-azure:daily-snapshot-2023.06.13-b7noa2ggroapnm9mfg_1oq/q-KH36L9TNyZZfLqIYmpvQ] failed to snapshot shard", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][snapshot][T#256]","log.logger":"org.elasticsearch.snapshots.SnapshotShardsService","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.common.util.concurrent.UncategorizedExecutionException","error.message":"Failed execution","error.stack_trace":"org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.FutureUtils.rethrowExecutionException(FutureUtils.java:80)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:72)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListenerDirectly$0(ListenableFuture.java:111)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:499)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:111)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:149)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:139)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.StepListener.onFailure(StepListener.java:63)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.GroupedActionListener.onFailure(GroupedActionListener.java:72)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$fileQueueListener$81(BlobStoreRepository.java:3163)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.doRun(ShardSnapshotTaskRunner.java:108)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:51)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:27)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:134)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write blob indices/4dODuAmWQqeKS71Frxi23w/0/__pgo2aKd_Rj2I4MHdtxzBQA\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:231)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:53)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:65)\n\t... 23 more\nCaused by: java.io.IOException: Unable to write blob indices/4dODuAmWQqeKS71Frxi23w/0/__pgo2aKd_Rj2I4MHdtxzBQA\n\tat org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:445)\n\tat org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:93)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:3518)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.lambda$doRun$0(ShardSnapshotTaskRunner.java:106)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t... 10 more\n\tSuppressed: java.io.IOException: Unable to write blob indices/4dODuAmWQqeKS71Frxi23w/0/__MhwMWmMeR6O3OOSAjJHKZg\n\t\t... 16 more\n\tCaused by: org.elasticsearch.snapshots.AbortedSnapshotException: aborted\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.checkAborted(BlobStoreRepository.java:3511)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.read(BlobStoreRepository.java:3504)\n\t\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore$2.read(AzureBlobStore.java:516)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$convertStreamToByteBuffer$22(AzureBlobStore.java:550)\n\t\tat reactor.core.publisher.MonoCallable.call(MonoCallable.java:92)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:410)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\t\tat reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:236)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)\n\t\tat reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)\n\t\tat reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)\n\t\tat reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)\n\t\tat reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)\n\t\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)\n\t\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)\n\t\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.azure.AzureClientProvider$1.lambda$decorateRunnable$0(AzureClientProvider.java:230)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:891)\n\t\t... 3 more\n\t\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\t\tat reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)\n\t\t\tat reactor.core.publisher.Mono.block(Mono.java:1707)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:464)\n\t\t\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\t\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:452)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:433)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:93)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:3518)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.lambda$doRun$0(ShardSnapshotTaskRunner.java:106)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.doRun(ShardSnapshotTaskRunner.java:108)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:51)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:27)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:134)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t\t... 3 more\nCaused by: org.elasticsearch.snapshots.AbortedSnapshotException: aborted\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.checkAborted(BlobStoreRepository.java:3511)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.read(BlobStoreRepository.java:3504)\n\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\tat org.elasticsearch.repositories.azure.AzureBlobStore$2.read(AzureBlobStore.java:516)\n\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$convertStreamToByteBuffer$22(AzureBlobStore.java:550)\n\tat reactor.core.publisher.MonoCallable.call(MonoCallable.java:92)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:410)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)\n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)\n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:236)\n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)\n\tat reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)\n\tat reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)\n\tat reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)\n\tat reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)\n\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)\n\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.azure.AzureClientProvider$1.lambda$decorateRunnable$0(AzureClientProvider.java:230)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:891)\n\t... 3 more\n\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\tat reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)\n\t\tat reactor.core.publisher.Mono.block(Mono.java:1707)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:464)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:452)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:433)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:93)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:3518)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.lambda$doRun$0(ShardSnapshotTaskRunner.java:106)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.doRun(ShardSnapshotTaskRunner.java:108)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:51)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:27)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:134)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t... 3 more\n"}
5 
{"@timestamp":"2023-06-13T09:10:57.762Z", "log.level": "WARN", "message":"cluster state applier task [ApplyCommitRequest{term=78, version=47979, sourceNode={elasticsearch-master-1}{LJ_xcrrVQNiNaQ_MWWyD5g}{0xFzWdPpRnyT0XuT19E_OA}{elasticsearch-master-1}{172.19.55.113}{172.19.55.113:9300}{dim}{8.7.0}{xpack.installed=true}}] took [38s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=78, version=47979, sourceNode={elasticsearch-master-1}{LJ_xcrrVQNiNaQ_MWWyD5g}{0xFzWdPpRnyT0XuT19E_OA}{elasticsearch-master-1}{172.19.55.113}{172.19.55.113:9300}{dim}{8.7.0}{xpack.installed=true}}]] took [0ms], [connecting to new nodes] took [0ms], [applying settings] took [0ms], [org.elasticsearch.repositories.RepositoriesService@d2e007c] took [0ms], [org.elasticsearch.indices.cluster.IndicesClusterStateService@38fac8a8] took [38017ms], [org.elasticsearch.script.ScriptService@1a7bcb43] took [0ms], [org.elasticsearch.xpack.ilm.IndexLifecycleService@24e1e7f7] took [0ms], [org.elasticsearch.snapshots.RestoreService@4a30c44b] took [0ms], [org.elasticsearch.ingest.IngestService@183c6f0f] took [3ms], [org.elasticsearch.action.ingest.IngestActionForwarder@7fde502d] took [0ms], [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$5776/0x0000000801edda68@1af458ea] took [0ms], [org.elasticsearch.indices.TimestampFieldMapperService@79c0b30c] took [0ms], [org.elasticsearch.tasks.TaskManager@545a80de] took [0ms], [org.elasticsearch.snapshots.SnapshotsService@6c51a1b7] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@725cbd1e] took [0ms], [org.elasticsearch.cluster.InternalClusterInfoService@69c816cd] took [0ms], [org.elasticsearch.snapshots.InternalSnapshotsInfoService@3e4f64f6] took [0ms], [org.elasticsearch.indices.SystemIndexManager@5686ca1b] took [0ms], [org.elasticsearch.xpack.monitoring.MonitoringTemplateRegistry@441509aa] took [0ms], [org.elasticsearch.xpack.autoscaling.capacity.nodeinfo.AutoscalingNodeInfoService$$Lambda$3680/0x0000000801961400@6dc2119b] took [0ms], [org.elasticsearch.xpack.ml.MlUpgradeModeActionFilter$$Lambda$3682/0x00000008018b7650@24ed69ed] took [0ms], [org.elasticsearch.xpack.ml.MlIndexTemplateRegistry@59defa22] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@c6426b0] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@f362f60] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@405b8d6a] took [0ms], [org.elasticsearch.xpack.ml.utils.persistence.ResultsPersisterService$$Lambda$3686/0x000000080196c428@6e7a70a9] took [0ms], [org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager@43069a70] took [0ms], [org.elasticsearch.xpack.ml.datafeed.DatafeedRunner$TaskRunner@7df5e05f] took [0ms], [org.elasticsearch.xpack.ml.inference.TrainedModelStatsService$$Lambda$3839/0x000000080198e028@61d90256] took [0ms], [org.elasticsearch.xpack.ml.inference.loadingservice.ModelLoadingService@5c1bc9fc] took [4ms], [org.elasticsearch.xpack.ml.process.MlMemoryTracker@52a8e628] took [0ms], [org.elasticsearch.xpack.ml.MlAssignmentNotifier@319a18cd] took [0ms], [org.elasticsearch.xpack.ml.MlAutoUpdateService@2fa1e494] took [0ms], [org.elasticsearch.xpack.ml.autoscaling.NodeAvailabilityZoneMapper@bf49183] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@2b6f1e9d] took [0ms], [org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentClusterService@78f94679] took [0ms], [org.elasticsearch.xpack.ml.autoscaling.MlAutoscalingDeciderService@5d1a1770] took [0ms], [org.elasticsearch.xpack.ml.MlInitializationService@25b5ce49] took [0ms], [org.elasticsearch.xpack.shutdown.NodeSeenService@6869077c] took [0ms], [org.elasticsearch.xpack.ccr.action.ShardFollowTaskCleaner@3b21671a] took [0ms], [org.elasticsearch.xpack.enrich.EnrichPolicyMaintenanceService@9d93489] took [0ms], [org.elasticsearch.xpack.transform.notifications.TransformAuditor$$Lambda$3872/0x00000008019a6998@295647d] took [0ms], [org.elasticsearch.xpack.transform.TransformClusterStateListener@66f971b] took [0ms], [org.elasticsearch.xpack.stack.StackTemplateRegistry@4f12138f] took [0ms], [org.elasticsearch.xpack.security.support.SecurityIndexManager@56255c53] took [0ms], [org.elasticsearch.xpack.security.support.SecurityIndexManager@1d99a9db] took [0ms], [org.elasticsearch.xpack.security.support.SecurityIndexManager@1a5ceb1b] took [0ms], [org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail@3fa3dd65] took [0ms], [org.elasticsearch.xpack.security.authc.TokenService$$Lambda$3911/0x0000000801a02a80@55936ad4] took [0ms], [org.elasticsearch.xpack.searchablesnapshots.cache.blob.BlobStoreCacheMaintenanceService@78edf146] took [0ms], [org.elasticsearch.xpack.searchablesnapshots.upgrade.SearchableSnapshotIndexMetadataUpgrader$$Lambda$4062/0x0000000801a33408@590453cb] took [0ms], [org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots$RepositoryUuidWatcher@170f57f5] took [0ms], [org.elasticsearch.xpack.watcher.support.WatcherIndexTemplateRegistry@2e49778a] took [0ms], [org.elasticsearch.xpack.watcher.WatcherLifeCycleService@72606313] took [0ms], [org.elasticsearch.xpack.watcher.WatcherIndexingListener@32012640] took [0ms], [org.elasticsearch.xpack.ilm.history.ILMHistoryTemplateRegistry@78eaa9cc] took [0ms], [org.elasticsearch.xpack.ilm.IndexLifecycleService@24e1e7f7] took [0ms], [org.elasticsearch.xpack.slm.history.SnapshotLifecycleTemplateRegistry@1dcab025] took [0ms], [org.elasticsearch.xpack.slm.SnapshotLifecycleService@1baa30b0] took [0ms], [org.elasticsearch.xpack.slm.SnapshotRetentionService@20a226a3] took [0ms], [org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingTemplateRegistry@d2b4fb2] took [0ms], [org.elasticsearch.xpack.fleet.FleetTemplateRegistry@f544be9] took [0ms], [org.elasticsearch.ingest.geoip.DatabaseNodeService$$Lambda$4154/0x00000008018d37d8@6ccfc802] took [0ms], [org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor@13f02bc4] took [3ms], [org.elasticsearch.cluster.metadata.SystemIndexMetadataUpgradeService@247794be] took [0ms], [org.elasticsearch.cluster.metadata.TemplateUpgradeService@6ac88b18] took [0ms], [org.elasticsearch.node.ResponseCollectorService@32cbee33] took [0ms], [org.elasticsearch.snapshots.SnapshotShardsService@2f861da] took [1ms], [org.elasticsearch.health.node.selection.HealthNodeTaskExecutor$$Lambda$4929/0x0000000801c5f4c0@3be49fd9] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@1f3f461d] took [0ms], [org.elasticsearch.xpack.ml.job.task.OpenJobPersistentTasksExecutor$$Lambda$4939/0x0000000801c5a000@3f17100f] took [0ms], [org.elasticsearch.xpack.ml.action.TransportStartDataFrameAnalyticsAction$TaskExecutor$$Lambda$4940/0x0000000801c5aa78@24f2716c] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@766eb077] took [0ms], [org.elasticsearch.xpack.ml.job.snapshot.upgrader.SnapshotUpgradeTaskExecutor$$Lambda$4941/0x0000000801c5b260@13841f09] took [0ms], [org.elasticsearch.persistent.PersistentTasksClusterService@1494f21e] took [0ms], [org.elasticsearch.shutdown.PluginShutdownService@23da98f1] took [0ms], [org.elasticsearch.cluster.coordination.MasterHistory@204c42ce] took [0ms], [org.elasticsearch.health.metadata.HealthMetadataService$$Lambda$4966/0x0000000801c65450@5fc16ab9] took [0ms], [org.elasticsearch.health.node.LocalHealthMonitor@3adbe51c] took [0ms], [org.elasticsearch.health.node.HealthInfoCache@7928d640] took [0ms], [org.elasticsearch.cluster.routing.DelayedAllocationService@d629ff7] took [0ms], [org.elasticsearch.indices.store.IndicesStore@35c2ca4d] took [1ms], [org.elasticsearch.persistent.PersistentTasksNodeService@481101b2] took [0ms], [org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentNodeService@44c5cef4] took [0ms], [org.elasticsearch.license.LicenseService@3c45d326] took [0ms], [org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator@41d47ac5] took [0ms], [org.elasticsearch.datastreams.UpdateTimeSeriesRangeService@45f461bd] took [0ms], [org.elasticsearch.xpack.core.async.AsyncTaskMaintenanceService@186b6306] took [0ms], [org.elasticsearch.gateway.GatewayService@973543] took [0ms], [org.elasticsearch.indices.recovery.PeerRecoverySourceService@62f775c6] took [0ms], [org.elasticsearch.reservedstate.service.FileSettingsService@46819ac] took [0ms], [org.elasticsearch.cluster.coordination.CoordinationDiagnosticsService@47d82974] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@1866fc30]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@4c33f5bf]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@7fa3438e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@20963ec0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@f3ec19e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@5cf353cb]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@3929d4e8]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@28d88d33]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@51139b87]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@17b8f765]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@6041ffee]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@c9c47c0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@419e3a77]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@11cdeefe]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@7e9cdda6]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$2@105027f2]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@20adfa0d]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@5e695af8]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@4600cd7e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@577e1dfb]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@184cf88b]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@ce6dfe0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@4ccd883f]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@6bfbd598]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@140ce0cc]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@2e2f3472]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@e83ddf8]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@3be18ea5]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@48181395]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@1f433240]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@fbf504d]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@389c63b9]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@30904b84]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@36c7aee4]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@430fbd49]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@18d59f3a]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@2aab3bcf]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@219010f0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@1f4b047e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@2bd364e2]]]] took [0ms]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:09.491Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:09.576Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:09.807Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:10.381Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:11.268Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:55.983Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:56.980Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:35.193Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:36.515Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:37.934Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:39.041Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:16:42.841Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:16:46.810Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:17:53.456Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:17:56.152Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:18:57.833Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:18:58.409Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:18:59.352Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:00.256Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:01.221Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:56.732Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:58.274Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:21:17.323Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
6

[org.elasticsearch.indices.cluster.IndicesClusterStateService@38fac8a8] took [38017ms] is the problem. Usually this means egregiously slow storage.

2 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.