while creating new Index shard acknowledgement fails, due to this mapping is not updated to the Index and status is yellow. Indexing fails due to ReceiveTimeoutTransportException. Also, Repo creation fails due to the same Exception.
Cluster Details:
Data size - 60 GB
Node count - 1
Indices count - 94
Heap committed - 5 GB
Node Hot Threads:
0.0% (0s out of 500ms) cpu usage by thread 'Attach Listener'
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
0.0% (0s out of 500ms) cpu usage by thread 'Wrapper-Control-Event-Monitor'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.tanukisoftware.wrapper.WrapperManager$3.run(WrapperManager.java:1074)
0.0% (0s out of 500ms) cpu usage by thread 'Wrapper-Connection'
10/10 snapshots sharing following 9 elements
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
java.net.SocketInputStream.read(SocketInputStream.java:224)
java.io.DataInputStream.readByte(DataInputStream.java:265)
org.tanukisoftware.wrapper.WrapperManager.handleBackend(WrapperManager.java:5891)
org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:6332)
java.lang.Thread.run(Thread.java:748)
0.0% (0s out of 500ms) cpu usage by thread 'DestroyJavaVM'
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
unique snapshot
0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[keepAlive/5.6.4]'
10/10 snapshots sharing following 8 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:999)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:84)
java.lang.Thread.run(Thread.java:748)
0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:536)
0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][clusterService#updateTask][T#1]'
10/10 snapshots sharing following 16 elements
org.elasticsearch.index.shard.IndexShard.updateRoutingEntry(IndexShard.java:380)
org.elasticsearch.indices.cluster.IndicesClusterStateService.updateShard(IndicesClusterStateService.java:557)
org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:521)
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:204)
org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814)
org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768)
org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587)
org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263)
org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247)
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.0% (0s out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][generic][T#7]'
10/10 snapshots sharing following 31 elements
sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:110)
java.nio.channels.FileChannel.open(FileChannel.java:287)
java.nio.channels.FileChannel.open(FileChannel.java:335)
org.elasticsearch.index.translog.Translog$$Lambda$1448/1714002180.open(Unknown Source)
org.elasticsearch.index.translog.ChannelFactory.open(ChannelFactory.java:33)
org.elasticsearch.index.translog.TranslogWriter.create(TranslogWriter.java:92)
org.elasticsearch.index.translog.Translog.createWriter(Translog.java:379)
org.elasticsearch.index.translog.Translog.<init>(Translog.java:203)
org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:272)
org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:160)
org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1602)
org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1584)
org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1027)
org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:987)
org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:360)
org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:90)
org.elasticsearch.index.shard.StoreRecovery$$Lambda$1444/521597373.run(Unknown Source)
org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257)
org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:88)
org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1236)
org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$1(IndexShard.java:1484)
org.elasticsearch.index.shard.IndexShard$$Lambda$1443/1274477608.run(Unknown Source)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][flush][T#1]'
10/10 snapshots sharing following 12 elements
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3006)
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3244)
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3207)
org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1576)
org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1062)
org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:777)
org.elasticsearch.index.shard.IndexShard$2.doRun(IndexShard.java:1748)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][refresh][T#2]'
10/10 snapshots sharing following 41 elements
sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225)
java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
java.nio.file.Files.newOutputStream(Files.java:216)
org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:413)
org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:409)
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
org.apache.lucene.store.RateLimitedFSDirectory.createOutput(RateLimitedFSDirectory.java:40)
org.apache.lucene.store.FilterDirectory.createOutput(FilterDirectory.java:73)
org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
org.apache.lucene.codecs.lucene53.Lucene53NormsConsumer.<init>(Lucene53NormsConsumer.java:46)
org.apache.lucene.codecs.lucene53.Lucene53NormsFormat.normsConsumer(Lucene53NormsFormat.java:77)
org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:299)
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:136)
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:444)
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539)
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653)
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:445)
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256)
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104)
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:909)
org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:632)
org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:690)
org.elasticsearch.index.IndexService.access$400(IndexService.java:92)
org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:832)
org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:743)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Please help us to identify the possible root cause.
You're running Elasticsearch in some strange wrapper thing it seems. This isn't a supported environment. You'll need to run Elasticsearch as described in the manual instead.
Sorry, it is not supported to run Elasticsearch within this wrapper. You'll either need to reproduce the problem without it, or contact the Tanuki developers for further assistance.
I have no other ideas for troubleshooting such an old version. Once you've upgraded, we can likely help further.
I understand your concern. since, our data size is huge. due to Indices compatibility, we are currently upgrading to 6.5.4 and so on in future.
we have some finding from analyzing hot Threads, Please help us to find the possible root cause.
15.6% (78.1ms out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][clusterService#updateTask][T#1]'
ClusterService.executeTask method is waiting for some time, due to WindowsNativeDispatcher.MoveFileEx0 . Also, we have noticed bulk request threads waiting because of WindowsNativeDispatcher.CreateFile0
18.8% (93.7ms out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][bulk][T#4]'
10/10 snapshots sharing following 60 elements
sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225)
java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
java.nio.file.Files.newOutputStream(Files.java:216)
org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:413)
org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:409)
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
org.apache.lucene.store.RateLimitedFSDirectory.createOutput(RateLimitedFSDirectory.java:40)
org.apache.lucene.store.FilterDirectory.createOutput(FilterDirectory.java:73)
org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:117)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)
org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)
org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:364)
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:398)
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1571)
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:662)
org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:606)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:504)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:557)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:492)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:146)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:115)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:70)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270)
org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924)
org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921)
org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933)
org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:107)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:92)
com.floragunn.searchguard.SearchGuardPlugin$4$1.messageReceived(SearchGuardPlugin.java:423)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
15.6% (78.1ms out of 500ms) cpu usage by thread 'elasticsearch[int_desktop-7jo7479][clusterService#updateTask][T#1]'
10/10 snapshots sharing following 45 elements
sun.nio.fs.WindowsNativeDispatcher.MoveFileEx0(Native Method)
sun.nio.fs.WindowsNativeDispatcher.MoveFileEx(WindowsNativeDispatcher.java:327)
sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:293)
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
java.nio.file.Files.move(Files.java:1395)
org.apache.logging.log4j.core.appender.rolling.action.FileRenameAction.moveFile(FileRenameAction.java:178)
org.apache.logging.log4j.core.appender.rolling.action.FileRenameAction.execute(FileRenameAction.java:121)
org.apache.logging.log4j.core.appender.rolling.action.FileRenameAction.execute(FileRenameAction.java:68)
org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:509)
org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:392)
org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:308)
org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:311)
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:388)
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
org.apache.logging.log4j.core.Logger.logMessage(Logger.java:153)
org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:222)
org.elasticsearch.common.logging.PrefixLogger.logMessage(PrefixLogger.java:102)
org.apache.logging.log4j.spi.AbstractLogger.log(AbstractLogger.java:2117)
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2040)
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1907)
org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1449)
org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:332)
org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230)
org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634)
org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612)
org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571)
org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263)
org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247)
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
9.4% (46.8ms out of 500ms) cpu usage by thread 'pool-2-thread-1'
10/10 snapshots sharing following 16 elements
sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225)
java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
java.nio.file.Files.newOutputStream(Files.java:216)
com.zoho.za.dae.elastic.ESHealthMonitor.fsHealthCheck(ESHealthMonitor.java:62)
com.zoho.za.dae.elastic.ESHealthMonitor.run(ESHealthMonitor.java:52)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.0% (0s out of 500ms) cpu usage by thread 'Wrapper-Connection'
10/10 snapshots sharing following 9 elements
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
java.net.SocketInputStream.read(SocketInputStream.java:224)
java.io.DataInputStream.readByte(DataInputStream.java:265)
org.tanukisoftware.wrapper.WrapperManager.handleBackend(WrapperManager.java:5891)
org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:6332)
java.lang.Thread.run(Thread.java:748)
from this it seems, due to disk slowness the threads are waiting that results in cluster event timeout.
Please let us know that the above finding could be the possible reason. Also, kindly share the other possible reasons (if any).
Seems like a reasonable guess although this code has changed beyond recognition in the 5½ years since 6.5.4 was released and I'm not planning on digging back through history to do a proper analysis.
You're using the third-party SearchGuard plugin too it seems, perhaps this could also be related? This hasn't been necessary for a very long time, security features were added to the core product ages ago.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.