Node failed to run

Elastic Stack Elasticsearch
1

Node has been restarted and then

[2022-04-23T05:04:36,452][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elastic-elasticsearch-data-2] fatal error in thread [elasticsearch[elastic-elasticsearch-data-2][generic][T#14]], exiting
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:214) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekCeil(SegmentTermsEnum.java:756) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.FrozenBufferedUpdates$TermDocsIterator.nextTerm(FrozenBufferedUpdates.java:578) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.FrozenBufferedUpdates.applyDocValuesUpdates(FrozenBufferedUpdates.java:256) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.FrozenBufferedUpdates.applyDocValuesUpdates(FrozenBufferedUpdates.java:201) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:171) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter.forceApply(IndexWriter.java:5658) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter.tryApply(IndexWriter.java:5567) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
fatal error in thread [elasticsearch[elastic-elasticsearch-data-2][generic][T#14]], exiting
	at org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$10(IndexWriter.java:2742) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter$EventQueue.processEventsInternal(IndexWriter.java:354) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter$EventQueue.processEvents(IndexWriter.java:344) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5426) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3376) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3770) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3728) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:2767) ~[elasticsearch-7.17.1.jar:7.17.1]
	at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:2075) ~[elasticsearch-7.17.1.jar:7.17.1]
	at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:526) ~[elasticsearch-7.17.1.jar:7.17.1]
	at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:483) ~[elasticsearch-7.17.1.jar:7.17.1]
	at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:119) ~[elasticsearch-7.17.1.jar:7.17.1]
	at org.elasticsearch.index.shard.IndexShard.recoverLocallyUpToGlobalCheckpoint(IndexShard.java:1833) ~[elasticsearch-7.17.1.jar:7.17.1]
	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:230) ~[elasticsearch-7.17.1.jar:7.17.1]
	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:76) ~[elasticsearch-7.17.1.jar:7.17.1]

anything I can do with this?

2

That usually indicates some kind of unrecoverable I/O error. Check your kernel message log (e.g. dmesg) for more details, and replace the faulty disk to fix it.

3

Okay, thank you.

I found it's a disk.

What can I do now? Can I find a particular file or dir that causes problems?

4

Not really, the only fix for a faulty disk is to replace it. Elasticsearch will recover the data it held from other nodes in the cluster.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.