Index Status goes Unknown after Dramatic Increase in Size

[Senthamarai_Kannan]

One of our ES indices go to Unknown status after the size increases dramatically. Usually the index size is about 300MB but it goes above 8GB and goes to Unknown state.

This is a single node instance hosted in a Windows VM

Would appreciate any help to resolve this.

[warkolm (Mark Walkom)]

There's not a heap of information to work with here.

What version are you on?
Where are you seeing this unknown state?
What do your Elasticsearch logs show?

[Senthamarai_Kannan]

We are using latest version 7.10.2

I see size growing in Head pluging and the plugin also shows Unknown status after some time.

Error Log has a number of errors.

[2021-02-03T02:06:30,164][WARN ][o.e.t.ThreadPool ] [node-1] failed to run scheduled task [org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@35e9a551] on thread pool [same]
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.index.IndexWriter.getFlushingBytes(IndexWriter.java:781) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.index.engine.InternalEngine.getWritingBytes(InternalEngine.java:622) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.index.shard.IndexShard.getWritingBytes(IndexShard.java:978) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.indices.IndexingMemoryController.getShardWritingBytes(IndexingMemoryController.java:182) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.runUnlocked(IndexingMemoryController.java:310) ~[elasticsearch-7.10.2.jar:7.10.2]

[2021-02-03T02:06:30,819][DEBUG][o.e.a.s.TransportSearchAction] [node-1] [QkUuXxWMQ_axo8-atRer0g][videoindex42][0]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[videoindex32, videoindex2, videoindex31, videoindex18, videoindex19, videoindex16, videoindex17, videoindex39, videoindex36, videoindex37, videoindex34, videoindex43, videoindex22, videoindex44, videoindex41, videoindex42, videoindex40, videoindex29, videoindex25, videoindex47, videoindex26, videoindex48, videoindex45, videoindex46, .security, .security-7], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=, routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":50,"query":{"bool":{"should":[{"match_all":{"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"sort":,"aggregations":{}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [node-1][10.1.13.5:9300][indices:data/read/search[phase/query]]
Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed
at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.index.engine.Engine.acquireSearcherSupplier(Engine.java:640) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.index.engine.Engine.acquireSearcher(Engine.java:688) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1272) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1261) ~[elasticsearch-7.10.2.jar:7.10.2]

[2021-02-03T02:06:32,931][DEBUG][o.e.a.s.TransportSearchAction] [node-1] [QkUuXxWMQ_axo8-atRer0g][videoindex42][0]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[videoindex42], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=, routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":50,"query":{"bool":{"should":[{"match_all":{"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"sort":,"aggregations":{}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [node-1][10.1.13.5:9300][indices:data/read/search[phase/query]]
Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed
at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.index.engine.Engine.acquireSearcherSupplier(Engine.java:640) ~[elasticsearch-7.10.2.jar:7.10.2]