Elasticsearch snapshots fail everyday

ramdas · June 12, 2023, 9:05am

Hi,

I am using elasticsearch 8.7 in out production cluster on Azure kubernetes platform which has 8 data and master nodes and almost 4TB per node disks being used to store our observability data. we have 699 indexes as of now and almost same number of primary and replica shards.

we are using azure blob storage for snapshots and they were being successfully done till last week since when below strange errors are causing them to fail everyday. can anybody help to solve it please?

INTERNAL_SERVER_ERROR: UncategorizedExecutionException[Failed execution]; nested: ExecutionException[java.io.IOException: Unable to write blob indices/cQV-28f_Q4G_xbuxagrf9g/0/__pr8s4d-PR2Cf6o_odhWGrA]; nested: IOException[Unable to write blob indices/cQV-28f_Q4G_xbuxagrf9g/0/__pr8s4d-PR2Cf6o_odhWGrA]; nested: AbortedSnapshotException[aborted]

warkolm · June 13, 2023, 12:50am

Is there more to the Elasticsearch logs you can share?

ramdas · June 14, 2023, 10:07am

Hi,

Thanks for your response. i am sharing logs in see related to snapshots

master node:

{"@timestamp":"2023-06-13T01:30:00.001Z", "log.level": "INFO", "message":"starting SLM retention snapshot cleanup task", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:00.001Z", "log.level": "INFO", "message":"snapshot lifecycle policy [daily-snapshot-everything] issuing create snapshot [daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotLifecycleTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:00.001Z", "log.level": "INFO", "message":"snapshot lifecycle policy job [daily-snapshot-everything-7] issued new snapshot creation for [daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa] successfully", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotLifecycleTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:00.274Z", "log.level": "INFO", "message":"SLM retention snapshot cleanup task complete", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][snapshot_meta][T#1896]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:30:03.854Z", "log.level": "INFO", "message":"snapshot [snapshot-repo-azure:daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa/0Av1n922RS-6er5nVpqS_g] started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.snapshots.SnapshotsService","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:57:40.930Z", "log.level":"ERROR", "message":"failed to create snapshot for snapshot lifecycle policy [daily-snapshot-everything]: org.elasticsearch.snapshots.SnapshotException: [snapshot-repo-azure:daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa] failed to create snapshot successfully, 18 out of 740 total shards failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][snapshot][T#475]","log.logger":"org.elasticsearch.xpack.slm.SnapshotLifecycleTask","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T01:57:40.930Z", "log.level": "INFO", "message":"snapshot [snapshot-repo-azure:daily-snapshot-2023.06.13-ffgjqwaiqlesru-sfop4oa/0Av1n922RS-6er5nVpqS_g] completed with state [PARTIAL]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-1][snapshot][T#475]","log.logger":"org.elasticsearch.snapshots.SnapshotsService","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"LJ_xcrrVQNiNaQ_MWWyD5g","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch"}

I also noticed that when i trigger snapshot...it initiates a cluster event for obvious reasons and when any of node does not process it within 10 seconds then master kicks that node out and it rejoins after some time. i suspect snapshot fails for this reason but i dont know how to fix kit.

i have 8 nodes with 24GB memory limits and 12GB heap size.

ramdas · June 14, 2023, 10:08am

Logs from Data node:

{"@timestamp":"2023-06-13T09:10:57.569Z", "log.level": "WARN", "message":"[[qa2-cust-eastus-logs-2023.06.13][0]][snapshot-repo-azure:daily-snapshot-2023.06.13-b7noa2ggroapnm9mfg_1oq/q-KH36L9TNyZZfLqIYmpvQ] failed to snapshot shard", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][snapshot][T#256]","log.logger":"org.elasticsearch.snapshots.SnapshotShardsService","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.common.util.concurrent.UncategorizedExecutionException","error.message":"Failed execution","error.stack_trace":"org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.FutureUtils.rethrowExecutionException(FutureUtils.java:80)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:72)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListenerDirectly$0(ListenableFuture.java:111)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:499)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:111)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:149)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:139)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.StepListener.onFailure(StepListener.java:63)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:97)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.support.GroupedActionListener.onFailure(GroupedActionListener.java:72)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$fileQueueListener$81(BlobStoreRepository.java:3163)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:223)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.doRun(ShardSnapshotTaskRunner.java:108)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:51)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:27)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:134)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write blob indices/4dODuAmWQqeKS71Frxi23w/0/__pgo2aKd_Rj2I4MHdtxzBQA\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:231)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:53)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:65)\n\t... 23 more\nCaused by: java.io.IOException: Unable to write blob indices/4dODuAmWQqeKS71Frxi23w/0/__pgo2aKd_Rj2I4MHdtxzBQA\n\tat org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:445)\n\tat org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:93)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:3518)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.lambda$doRun$0(ShardSnapshotTaskRunner.java:106)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t... 10 more\n\tSuppressed: java.io.IOException: Unable to write blob indices/4dODuAmWQqeKS71Frxi23w/0/__MhwMWmMeR6O3OOSAjJHKZg\n\t\t... 16 more\n\tCaused by: org.elasticsearch.snapshots.AbortedSnapshotException: aborted\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.checkAborted(BlobStoreRepository.java:3511)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.read(BlobStoreRepository.java:3504)\n\t\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore$2.read(AzureBlobStore.java:516)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$convertStreamToByteBuffer$22(AzureBlobStore.java:550)\n\t\tat reactor.core.publisher.MonoCallable.call(MonoCallable.java:92)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:410)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\t\tat reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:236)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)\n\t\tat reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)\n\t\tat reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)\n\t\tat reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)\n\t\tat reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)\n\t\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)\n\t\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)\n\t\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.azure.AzureClientProvider$1.lambda$decorateRunnable$0(AzureClientProvider.java:230)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:891)\n\t\t... 3 more\n\t\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\t\tat reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)\n\t\t\tat reactor.core.publisher.Mono.block(Mono.java:1707)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:464)\n\t\t\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\t\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:452)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:433)\n\t\t\tat org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:93)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:3518)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.lambda$doRun$0(ShardSnapshotTaskRunner.java:106)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.doRun(ShardSnapshotTaskRunner.java:108)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:51)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:27)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:134)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\t\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t\t... 3 more\nCaused by: org.elasticsearch.snapshots.AbortedSnapshotException: aborted\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.checkAborted(BlobStoreRepository.java:3511)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository$12.read(BlobStoreRepository.java:3504)\n\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\tat org.elasticsearch.repositories.azure.AzureBlobStore$2.read(AzureBlobStore.java:516)\n\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$convertStreamToByteBuffer$22(AzureBlobStore.java:550)\n\tat reactor.core.publisher.MonoCallable.call(MonoCallable.java:92)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:410)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)\n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)\n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:236)\n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)\n\tat reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)\n\tat reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)\n\tat reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)\n\tat reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)\n\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)\n\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.azure.AzureClientProvider$1.lambda$decorateRunnable$0(AzureClientProvider.java:230)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:891)\n\t... 3 more\n\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\tat reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)\n\t\tat reactor.core.publisher.Mono.block(Mono.java:1707)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:464)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:452)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:433)\n\t\tat org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:93)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:3518)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.lambda$doRun$0(ShardSnapshotTaskRunner.java:106)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.repositories.blobstore.ShardSnapshotTaskRunner$FileSnapshotTask.doRun(ShardSnapshotTaskRunner.java:108)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:51)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.PrioritizedThrottledTaskRunner$TaskWrapper.onResponse(PrioritizedThrottledTaskRunner.java:27)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:134)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\t\tat org.elasticsearch.server@8.7.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t... 3 more\n"}

ramdas · June 14, 2023, 10:08am

{"@timestamp":"2023-06-13T09:10:57.762Z", "log.level": "WARN", "message":"cluster state applier task [ApplyCommitRequest{term=78, version=47979, sourceNode={elasticsearch-master-1}{LJ_xcrrVQNiNaQ_MWWyD5g}{0xFzWdPpRnyT0XuT19E_OA}{elasticsearch-master-1}{172.19.55.113}{172.19.55.113:9300}{dim}{8.7.0}{xpack.installed=true}}] took [38s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=78, version=47979, sourceNode={elasticsearch-master-1}{LJ_xcrrVQNiNaQ_MWWyD5g}{0xFzWdPpRnyT0XuT19E_OA}{elasticsearch-master-1}{172.19.55.113}{172.19.55.113:9300}{dim}{8.7.0}{xpack.installed=true}}]] took [0ms], [connecting to new nodes] took [0ms], [applying settings] took [0ms], [org.elasticsearch.repositories.RepositoriesService@d2e007c] took [0ms], [org.elasticsearch.indices.cluster.IndicesClusterStateService@38fac8a8] took [38017ms], [org.elasticsearch.script.ScriptService@1a7bcb43] took [0ms], [org.elasticsearch.xpack.ilm.IndexLifecycleService@24e1e7f7] took [0ms], [org.elasticsearch.snapshots.RestoreService@4a30c44b] took [0ms], [org.elasticsearch.ingest.IngestService@183c6f0f] took [3ms], [org.elasticsearch.action.ingest.IngestActionForwarder@7fde502d] took [0ms], [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$5776/0x0000000801edda68@1af458ea] took [0ms], [org.elasticsearch.indices.TimestampFieldMapperService@79c0b30c] took [0ms], [org.elasticsearch.tasks.TaskManager@545a80de] took [0ms], [org.elasticsearch.snapshots.SnapshotsService@6c51a1b7] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@725cbd1e] took [0ms], [org.elasticsearch.cluster.InternalClusterInfoService@69c816cd] took [0ms], [org.elasticsearch.snapshots.InternalSnapshotsInfoService@3e4f64f6] took [0ms], [org.elasticsearch.indices.SystemIndexManager@5686ca1b] took [0ms], [org.elasticsearch.xpack.monitoring.MonitoringTemplateRegistry@441509aa] took [0ms], [org.elasticsearch.xpack.autoscaling.capacity.nodeinfo.AutoscalingNodeInfoService$$Lambda$3680/0x0000000801961400@6dc2119b] took [0ms], [org.elasticsearch.xpack.ml.MlUpgradeModeActionFilter$$Lambda$3682/0x00000008018b7650@24ed69ed] took [0ms], [org.elasticsearch.xpack.ml.MlIndexTemplateRegistry@59defa22] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@c6426b0] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@f362f60] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@405b8d6a] took [0ms], [org.elasticsearch.xpack.ml.utils.persistence.ResultsPersisterService$$Lambda$3686/0x000000080196c428@6e7a70a9] took [0ms], [org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager@43069a70] took [0ms], [org.elasticsearch.xpack.ml.datafeed.DatafeedRunner$TaskRunner@7df5e05f] took [0ms], [org.elasticsearch.xpack.ml.inference.TrainedModelStatsService$$Lambda$3839/0x000000080198e028@61d90256] took [0ms], [org.elasticsearch.xpack.ml.inference.loadingservice.ModelLoadingService@5c1bc9fc] took [4ms], [org.elasticsearch.xpack.ml.process.MlMemoryTracker@52a8e628] took [0ms], [org.elasticsearch.xpack.ml.MlAssignmentNotifier@319a18cd] took [0ms], [org.elasticsearch.xpack.ml.MlAutoUpdateService@2fa1e494] took [0ms], [org.elasticsearch.xpack.ml.autoscaling.NodeAvailabilityZoneMapper@bf49183] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@2b6f1e9d] took [0ms], [org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentClusterService@78f94679] took [0ms], [org.elasticsearch.xpack.ml.autoscaling.MlAutoscalingDeciderService@5d1a1770] took [0ms], [org.elasticsearch.xpack.ml.MlInitializationService@25b5ce49] took [0ms], [org.elasticsearch.xpack.shutdown.NodeSeenService@6869077c] took [0ms], [org.elasticsearch.xpack.ccr.action.ShardFollowTaskCleaner@3b21671a] took [0ms], [org.elasticsearch.xpack.enrich.EnrichPolicyMaintenanceService@9d93489] took [0ms], [org.elasticsearch.xpack.transform.notifications.TransformAuditor$$Lambda$3872/0x00000008019a6998@295647d] took [0ms], [org.elasticsearch.xpack.transform.TransformClusterStateListener@66f971b] took [0ms], [org.elasticsearch.xpack.stack.StackTemplateRegistry@4f12138f] took [0ms], [org.elasticsearch.xpack.security.support.SecurityIndexManager@56255c53] took [0ms], [org.elasticsearch.xpack.security.support.SecurityIndexManager@1d99a9db] took [0ms], [org.elasticsearch.xpack.security.support.SecurityIndexManager@1a5ceb1b] took [0ms], [org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail@3fa3dd65] took [0ms], [org.elasticsearch.xpack.security.authc.TokenService$$Lambda$3911/0x0000000801a02a80@55936ad4] took [0ms], [org.elasticsearch.xpack.searchablesnapshots.cache.blob.BlobStoreCacheMaintenanceService@78edf146] took [0ms], [org.elasticsearch.xpack.searchablesnapshots.upgrade.SearchableSnapshotIndexMetadataUpgrader$$Lambda$4062/0x0000000801a33408@590453cb] took [0ms], [org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots$RepositoryUuidWatcher@170f57f5] took [0ms], [org.elasticsearch.xpack.watcher.support.WatcherIndexTemplateRegistry@2e49778a] took [0ms], [org.elasticsearch.xpack.watcher.WatcherLifeCycleService@72606313] took [0ms], [org.elasticsearch.xpack.watcher.WatcherIndexingListener@32012640] took [0ms], [org.elasticsearch.xpack.ilm.history.ILMHistoryTemplateRegistry@78eaa9cc] took [0ms], [org.elasticsearch.xpack.ilm.IndexLifecycleService@24e1e7f7] took [0ms], [org.elasticsearch.xpack.slm.history.SnapshotLifecycleTemplateRegistry@1dcab025] took [0ms], [org.elasticsearch.xpack.slm.SnapshotLifecycleService@1baa30b0] took [0ms], [org.elasticsearch.xpack.slm.SnapshotRetentionService@20a226a3] took [0ms], [org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingTemplateRegistry@d2b4fb2] took [0ms], [org.elasticsearch.xpack.fleet.FleetTemplateRegistry@f544be9] took [0ms], [org.elasticsearch.ingest.geoip.DatabaseNodeService$$Lambda$4154/0x00000008018d37d8@6ccfc802] took [0ms], [org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor@13f02bc4] took [3ms], [org.elasticsearch.cluster.metadata.SystemIndexMetadataUpgradeService@247794be] took [0ms], [org.elasticsearch.cluster.metadata.TemplateUpgradeService@6ac88b18] took [0ms], [org.elasticsearch.node.ResponseCollectorService@32cbee33] took [0ms], [org.elasticsearch.snapshots.SnapshotShardsService@2f861da] took [1ms], [org.elasticsearch.health.node.selection.HealthNodeTaskExecutor$$Lambda$4929/0x0000000801c5f4c0@3be49fd9] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@1f3f461d] took [0ms], [org.elasticsearch.xpack.ml.job.task.OpenJobPersistentTasksExecutor$$Lambda$4939/0x0000000801c5a000@3f17100f] took [0ms], [org.elasticsearch.xpack.ml.action.TransportStartDataFrameAnalyticsAction$TaskExecutor$$Lambda$4940/0x0000000801c5aa78@24f2716c] took [0ms], [org.elasticsearch.xpack.ml.notifications.AbstractMlAuditor$$Lambda$1896/0x00000008015ba1d0@766eb077] took [0ms], [org.elasticsearch.xpack.ml.job.snapshot.upgrader.SnapshotUpgradeTaskExecutor$$Lambda$4941/0x0000000801c5b260@13841f09] took [0ms], [org.elasticsearch.persistent.PersistentTasksClusterService@1494f21e] took [0ms], [org.elasticsearch.shutdown.PluginShutdownService@23da98f1] took [0ms], [org.elasticsearch.cluster.coordination.MasterHistory@204c42ce] took [0ms], [org.elasticsearch.health.metadata.HealthMetadataService$$Lambda$4966/0x0000000801c65450@5fc16ab9] took [0ms], [org.elasticsearch.health.node.LocalHealthMonitor@3adbe51c] took [0ms], [org.elasticsearch.health.node.HealthInfoCache@7928d640] took [0ms], [org.elasticsearch.cluster.routing.DelayedAllocationService@d629ff7] took [0ms], [org.elasticsearch.indices.store.IndicesStore@35c2ca4d] took [1ms], [org.elasticsearch.persistent.PersistentTasksNodeService@481101b2] took [0ms], [org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentNodeService@44c5cef4] took [0ms], [org.elasticsearch.license.LicenseService@3c45d326] took [0ms], [org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator@41d47ac5] took [0ms], [org.elasticsearch.datastreams.UpdateTimeSeriesRangeService@45f461bd] took [0ms], [org.elasticsearch.xpack.core.async.AsyncTaskMaintenanceService@186b6306] took [0ms], [org.elasticsearch.gateway.GatewayService@973543] took [0ms], [org.elasticsearch.indices.recovery.PeerRecoverySourceService@62f775c6] took [0ms], [org.elasticsearch.reservedstate.service.FileSettingsService@46819ac] took [0ms], [org.elasticsearch.cluster.coordination.CoordinationDiagnosticsService@47d82974] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@1866fc30]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@4c33f5bf]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@7fa3438e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@20963ec0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@f3ec19e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@5cf353cb]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@3929d4e8]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@28d88d33]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@51139b87]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@17b8f765]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@6041ffee]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@c9c47c0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@419e3a77]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@11cdeefe]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@7e9cdda6]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$2@105027f2]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@20adfa0d]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@5e695af8]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@4600cd7e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@577e1dfb]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@184cf88b]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@ce6dfe0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@4ccd883f]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@6bfbd598]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@140ce0cc]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@2e2f3472]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@e83ddf8]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@3be18ea5]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@48181395]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@1f433240]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@fbf504d]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@389c63b9]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@30904b84]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@36c7aee4]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@430fbd49]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@18d59f3a]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@2aab3bcf]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@219010f0]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@1f4b047e]]]] took [0ms], [ClusterStateObserver[ObservingContext[ContextPreservingListener[org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2@2bd364e2]]]] took [0ms]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:09.491Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:09.576Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:09.807Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:10.381Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:11.268Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:55.983Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:12:56.980Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:35.193Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:36.515Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:37.934Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:14:39.041Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:16:42.841Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:16:46.810Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:17:53.456Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:17:56.152Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:18:57.833Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:18:58.409Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:18:59.352Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:00.256Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:01.221Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:56.732Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:19:58.274Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-06-13T09:21:17.323Z", "log.level": "WARN", "message":"Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","trace.id":"d3210be4832c460baec05afd93d76696","elasticsearch.cluster.uuid":"owBcUi5vSDORDcxfpHp3xw","elasticsearch.node.id":"roXLkoF_Szyv0p8kCVUSPA","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}

DavidTurner · June 14, 2023, 11:23am

[org.elasticsearch.indices.cluster.IndicesClusterStateService@38fac8a8] took [38017ms] is the problem. Usually this means egregiously slow storage.

system · July 12, 2023, 11:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed to write shard level snapshot Elasticsearch snapshot-and-restore	3	686	October 20, 2021
Elasticsearch snapshot is failing due to access denied exception Elasticsearch snapshot-and-restore	16	991	February 13, 2024
Elasticsearch snapshot partialy fails due to "index shard snapshot failed exception" Elasticsearch	5	2425	December 10, 2019
Snapshots occasionally fail Elasticsearch snapshot-and-restore	1	519	February 28, 2022
Snapshot failing for 1 shard Elasticsearch slm-snapshot-lifecycle-management	4	1778	March 8, 2021

Elasticsearch snapshots fail everyday

Related Topics