Hi, i have a problem with elasticsearch on kubernetes.
here the problem:
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2021-08-06T14:28:31.995Z",
"failed_allocation_attempts" : 3,
"details" : """failed shard on node [wsTDaACvQTOQdBheyzOGdg]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [inetsi-2021.08.04][2]: Recovery failed from {elasticsearch-master-1}{LY-jylbCRGyw7p73VAu5Xg}{zmy_D534S6i-GxawJsuTPw}{10.42.9.38}{10.42.9.38:9300}{dim}{xpack.installed=true} into {elasticsearch-master-2}{wsTDaACvQTOQdBheyzOGdg}{IpvOjjpYSx-YocAsjKi1vA}{10.42.7.60}{10.42.7.60:9300}{dim}{xpack.installed=true}
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.handleException(PeerRecoveryTargetService.java:632)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1174)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1174)
at org.elasticsearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:313)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.lang.Thread.run(Thread.java:831)
Caused by: org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-1][10.42.9.38:9300][internal:index/shard/recovery/start_recovery]
Caused by: org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-1][10.42.9.38:9300][indices:admin/seq_no/retention_lease_sync[p]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of primary operation [coordinating_and_primary_bytes=2206343514, replica_bytes=1370085522, all_bytes=3576429036, primary_operation_bytes=0, max_coordinating_and_primary_bytes=3221225472]
at org.elasticsearch.index.IndexingPressure.markPrimaryOperationStarted(IndexingPressure.java:120)
at org.elasticsearch.action.support.replication.TransportWriteAction.checkPrimaryLimits(TransportWriteAction.java:111)
at org.elasticsearch.action.support.replication.TransportWriteAction.checkPrimaryLimits(TransportWriteAction.java:50)
at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:288)
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:230)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:301)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61)
at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:792)
at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:114)
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:736)
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:147)
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.lambda$sendRequest$0(SecurityServerTransportInterceptor.java:99)
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:139)
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:99)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:650)
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:701)
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:687)
at org.elasticsearch.index.seqno.RetentionLeaseSyncAction.sync(RetentionLeaseSyncAction.java:106)
at org.elasticsearch.index.seqno.RetentionLeaseSyncer.sync(RetentionLeaseSyncer.java:38)
at org.elasticsearch.index.shard.IndexShard.lambda$new$0(IndexShard.java:361)
at org.elasticsearch.index.seqno.ReplicationTracker.cloneRetentionLease(ReplicationTracker.java:341)
at org.elasticsearch.index.seqno.ReplicationTracker.cloneLocalPeerRecoveryRetentionLease(ReplicationTracker.java:506)
at org.elasticsearch.index.shard.IndexShard.cloneLocalPeerRecoveryRetentionLease(IndexShard.java:2797)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$createRetentionLease$29(RecoverySourceHandler.java:575)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$runUnderPrimaryPermit$19(RecoverySourceHandler.java:367)
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:82)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.runUnderPrimaryPermit(RecoverySourceHandler.java:345)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.createRetentionLease(RecoverySourceHandler.java:563)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$phase1$23(RecoverySourceHandler.java:520)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:133)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:52)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29)
at org.elasticsearch.indices.recovery.MultiChunkTransfer.onCompleted(MultiChunkTransfer.java:142)
at org.elasticsearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:107)
at org.elasticsearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:67)
at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:97)
at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:85)
at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:73)
at org.elasticsearch.indices.recovery.MultiChunkTransfer.addItem(MultiChunkTransfer.java:78)
at org.elasticsearch.indices.recovery.MultiChunkTransfer.lambda$handleItems$3(MultiChunkTransfer.java:113)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:387)
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:387)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:387)
at org.elasticsearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:133)
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:387)
at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1164)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1164)
at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:287)
at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:275)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.lang.Thread.run(Thread.java:831)
elasticsearch is configured in this way:
esJavaOpts: "-Xmx30g -Xms30g"
3 nodes with master-data roles
3 shards for index with 1 replica
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
35 2.8gb 6.5gb 91.3gb 97.9gb 6 10.42.7.60 10.42.7.60 elasticsearch-master-2
33 2.6gb 7.5gb 90.4gb 97.9gb 7 10.42.9.38 10.42.9.38 elasticsearch-master-1
35 2.5gb 6.3gb 91.5gb 97.9gb 6 10.42.10.139 10.42.10.139 elasticsearch-master-0
3 UNASSIGNED
I don't understand why it gives me this error. the server is not under pressure in RAM or CPU. The indexs is not so big:
inetsi-2021.08.05 2 p STARTED 7000501 451.3mb 10.42.7.60 elasticsearch-master-2
inetsi-2021.08.05 2 r INITIALIZING 10.42.9.38 elasticsearch-master-1
inetsi-2021.08.05 1 r STARTED 4994013 340.8mb 10.42.10.139 elasticsearch-master-0
inetsi-2021.08.05 1 p STARTED 5476452 351.5mb 10.42.9.38 elasticsearch-master-1
inetsi-2021.08.05 0 p STARTED 5670786 363.2mb 10.42.10.139 elasticsearch-master-0
inetsi-2021.08.05 0 r STARTED 5272284 357mb 10.42.9.38 elasticsearch-master-1
What do i have to do, to fix this problem?
thanks
Cristian