Elasticsearch 7.8.1， shards allocation failed

wzhanw · December 22, 2020, 9:03am

Hi all,
I'm running the ealsticsearch 7.8.1 single node cluster on my server, I got a critical error when the logstash insert document to the es.
I tried to clean up the data directory and restarted the es, the error still happened. I didn't use any nfs for the storage. only one node with local disk storage.
Could someone help to check what's wrong here? Thank you.

[2020-12-22T01:39:53,007][WARN ][o.e.c.r.a.AllocationService] [master_dlw14.aus.mylabs.hello.com]failing shard [failed shard, shard [spark-task-cff1e1f9-273d-42a9-a78b-4cfcccf5fd6c-2020.12.22][0], node[943zkuXbQwyBMxKB47pSXg], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=4nxHv1UNS2ijreXuKCT3vw], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-12-22T07:39:52.195Z], failed_attempts[4], failed_nodes[[943zkuXbQwyBMxKB47pSXg]], delayed=false, details[failed shard on node [943zkuXbQwyBMxKB47pSXg]: shard failure, reason [lucene commit failed], failure IllegalStateException[iterator has already been used: docID=-1]], allocation_status[fetching_shard_data]], message [shard failure, reason [lucene commit failed]], failure [IllegalStateException[iterator has already been used: docID=-1]], markAsStale [true]]
java.lang.IllegalStateException: iterator has already been used: docID=-1
        at org.apache.lucene.index.SingletonSortedSetDocValues.<init>(SingletonSortedSetDocValues.java:37) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocValues.singleton(DocValues.java:271) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.doAddSortedField(Lucene80DocValuesConsumer.java:603) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.addSortedSetField(Lucene80DocValuesConsumer.java:746) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:129) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:221) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:263) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:138) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:555) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:722) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3200) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3445) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:2476) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:512) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:471) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:132) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1604) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:98) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery$$Lambda$2707/0000000000000000.get(Unknown Source) ~[?:?]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:325) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:96) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1877) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.IndexShard$$Lambda$2702/0000000000000000.accept(Unknown Source) ~[?:?]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.1.jar:7.8.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

 cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/

/openjdk/jre/11.0.8.10/linux-ppc64le/bin/java -version
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.21.0, JRE 11 Linux ppc64le-64-Bit Compressed References 20200715_650 (JIT enabled, AOT enabled)
OpenJ9   - 34cf4c075
OMR      - 113e54219
JCL      - 95bb504fbb based on jdk-11.0.8+10)

warkolm · December 22, 2020, 9:26am

Can you elaborate more on this, what exactly did you do?

wzhanw · December 23, 2020, 12:18am

Thank you for your quick reply, I removed the whole data directory(including master data) and all log files.

warkolm · December 23, 2020, 12:46am

Ok. You should never delete Elasticsearch files directly on the filesystem. That is guaranteed to create issues. Always use the delete API.

You might need to delete that index, using the API, and reinsert the data.

wzhanw · December 23, 2020, 7:49am

I stopped the ES first before removing the files. I think all data should be stored in the data directory after I cleaned them, then start it, it will work as a fresh install.

warkolm · December 24, 2020, 5:23am

DO NOT delete the files, even if Elasticsearch is not running. ALWAYS use the API.

system · January 21, 2021, 5:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.