Elasticsearch 7.8.1， shards allocation failed

Elastic Stack Elasticsearch
1

Hi all,
I'm running the ealsticsearch 7.8.1 single node cluster on my server, I got a critical error when the logstash insert document to the es.
I tried to clean up the data directory and restarted the es, the error still happened. I didn't use any nfs for the storage. only one node with local disk storage.
Could someone help to check what's wrong here? Thank you.

[2020-12-22T01:39:53,007][WARN ][o.e.c.r.a.AllocationService] [master_dlw14.aus.mylabs.hello.com]failing shard [failed shard, shard [spark-task-cff1e1f9-273d-42a9-a78b-4cfcccf5fd6c-2020.12.22][0], node[943zkuXbQwyBMxKB47pSXg], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=4nxHv1UNS2ijreXuKCT3vw], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-12-22T07:39:52.195Z], failed_attempts[4], failed_nodes[[943zkuXbQwyBMxKB47pSXg]], delayed=false, details[failed shard on node [943zkuXbQwyBMxKB47pSXg]: shard failure, reason [lucene commit failed], failure IllegalStateException[iterator has already been used: docID=-1]], allocation_status[fetching_shard_data]], message [shard failure, reason [lucene commit failed]], failure [IllegalStateException[iterator has already been used: docID=-1]], markAsStale [true]]
java.lang.IllegalStateException: iterator has already been used: docID=-1
        at org.apache.lucene.index.SingletonSortedSetDocValues.<init>(SingletonSortedSetDocValues.java:37) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocValues.singleton(DocValues.java:271) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.doAddSortedField(Lucene80DocValuesConsumer.java:603) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.addSortedSetField(Lucene80DocValuesConsumer.java:746) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:129) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:221) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:263) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:138) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:555) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:722) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3200) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3445) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
        at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:2476) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:512) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:471) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:132) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1604) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:98) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery$$Lambda$2707/0000000000000000.get(Unknown Source) ~[?:?]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:325) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:96) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1877) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.index.shard.IndexShard$$Lambda$2702/0000000000000000.accept(Unknown Source) ~[?:?]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.1.jar:7.8.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

 cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/

/openjdk/jre/11.0.8.10/linux-ppc64le/bin/java -version
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.21.0, JRE 11 Linux ppc64le-64-Bit Compressed References 20200715_650 (JIT enabled, AOT enabled)
OpenJ9   - 34cf4c075
OMR      - 113e54219
JCL      - 95bb504fbb based on jdk-11.0.8+10)
2

Can you elaborate more on this, what exactly did you do?

1 Like
3

Thank you for your quick reply, I removed the whole data directory(including master data) and all log files.

4

Ok. You should never delete Elasticsearch files directly on the filesystem. That is guaranteed to create issues. Always use the delete API.

You might need to delete that index, using the API, and reinsert the data.

1 Like
5

I stopped the ES first before removing the files. I think all data should be stored in the data directory after I cleaned them, then start it, it will work as a fresh install.

6

DO NOT delete the files, even if Elasticsearch is not running. ALWAYS use the API.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.