CorruptIndexException: checksum status indeterminate

tropically · November 9, 2021, 9:49pm

Hi
We recently had an outage on our openshift clusters.
In our Test environment the pods came back up with warnings.

The log file shows the following

Suppressed: org.apache.lucene.index.CorruptIndexException: checksum status indeterminate: remaining=0, please run checkindex for more details (resource=BufferedChecksumIndexInput(MMapIndexInput
(path="/usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe")))

however when we go that that directory the file is not present.

ls -ltr /usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe

ls: cannot access /usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe: No such file or directory

sh-4.2$ cd  /usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/        


sh-4.2$ ls
_0.cfe  _87d.dii  _87d.fdx  _87d_Lucene50_0.doc  _87d_Lucene70_0.dvd  _bjm.dim  _bjm.fnm             _bjm_Lucene50_0.tim  _bjm_Lucene70_0.dvm  _bjn.si   _bjo.si   _bjp.si   segments_4
_0.cfs  _87d.dim  _87d.fnm  _87d_Lucene50_0.tim  _87d_Lucene70_0.dvm  _bjm.fdt  _bjm.si              _bjm_Lucene50_0.tip  _bjn.cfe             _bjo.cfe  _bjp.cfe  _bjq.fdt  write.lock
_0.si   _87d.fdt  _87d.si   _87d_Lucene50_0.tip  _bjm.dii             _bjm.fdx  _bjm_Lucene50_0.doc  _bjm_Lucene70_0.dvd  _bjn.cfs             _bjo.cfs  _bjp.cfs  _bjq.fdx
sh-4.2$

How do we resolve this error?

TIA

tropically · November 9, 2021, 10:13pm

I ran check index and have the following message.

sh-4.2$ /opt/jdk-15.0.1+9/bin/java -cp /usr/share/elasticsearch/lib/lucene-core-7.7.3.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe 

Opening index @ /usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe

Exception in thread "main" org.apache.lucene.index.IndexNotFoundException: no segments* file found in MMapDirectory@/usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d6f64b1: files: [write.lock]
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:519)
        at org.apache.lucene.index.CheckIndex.doCheck(CheckIndex.java:2809)
        at org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:2711)
        at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2637)

DavidTurner · November 9, 2021, 10:13pm

That isn't the full error, and the details you have elided are important to answer your question. Would you share the complete log please?

(For the original error I mean, the CheckIndex output isn't useful here)

tropically · November 9, 2021, 10:16pm

It didn't allow me to paste the whole error due to size limits.
Uploading in parts


[2021-11-09T18:52:51,992][WARN ][o.e.g.G.InternalReplicaShardAllocator] [elasticsearch-master-0] [jaeger-span-2021-11-09][3]: failed to list shard for shard_store on node [ZIxGRmm0Tkmxlmxe3VhtNg]
org.elasticsearch.action.FailedNodeException: Failed node [ZIxGRmm0Tkmxlmxe3VhtNg]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:236) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:151) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:210) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1126) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1011) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1009) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1001) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:950) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) [transport-netty4-client-6.8.15.jar:6.8.15]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][10.129.2.25:9300][internal:cluster/nodes/indices/shard/store[n]]
Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[jaeger-span-2021-11-09][3]]
	at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:116) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:62) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:778) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) ~[?:?]
Caused by: org.apache.lucene.index.CorruptIndexException: failed engine (reason: [refresh failed source[schedule]]) (resource=preexisting_corruption)
	at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:599) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:442) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:165) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:114) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:62) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:778) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) ~[?:?]
Caused by: java.io.IOException: failed engine (reason: [refresh failed source[schedule]])
	at org.elasticsearch.index.engine.Engine.failEngine(Engine.java:1192) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1688) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1658) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:969) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:745) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService.access$500(IndexService.java:100) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:871) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) ~[elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) ~[?:?]

tropically · November 9, 2021, 10:17pm

Caused by: org.apache.lucene.index.CorruptIndexException: Problem reading index from store(ByteSizeCachingDirectory(MMapDirectory@/usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@ec6e853)) (resource=store(ByteSizeCachingDirectory(MMapDirectory@/usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@ec6e853)))
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:141) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:525) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:332) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:307) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1677) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1658) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:969) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:745) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService.access$500(IndexService.java:100) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:871) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) ~[elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) ~[?:?]
Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe")
	at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:108) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:100) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:525) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:332) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:307) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1677) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1658) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:969) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:745) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService.access$500(IndexService.java:100) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:871) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) ~[elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) ~[?:?]

tropically · November 9, 2021, 10:17pm

	Suppressed: org.apache.lucene.index.CorruptIndexException: checksum status indeterminate: remaining=0, please run checkindex for more details (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/2ubIJUqYTHOFOvKofHDl7Q/3/index/_f7l.cfe")))
		at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:454) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:126) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:100) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:525) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:332) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:307) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
		at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1677) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1658) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:969) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:745) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.index.IndexService.access$500(IndexService.java:100) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:871) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144) ~[elasticsearch-6.8.15.jar:6.8.15]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) ~[elasticsearch-6.8.15.jar:6.8.15]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.ru

DavidTurner · November 9, 2021, 10:23pm

I think this indicates that the files that make up this shard were changed by something other than Elasticsearch. However this is only about a replica, the primary should be unaffected and Elasticsearch will rebuild the replica in due course.

You'll need to work out why the data the Elasticsearch read wasn't the data it originally wrote, however. That will eventually cause you a bigger problem.

tropically · November 10, 2021, 3:38am

Hi David
How does elasticsearch rebuild these replica's?

If I needed to fix this prior to elasticsearch rebuilding, is that possible ? If so what would the steps be?

regards.

DavidTurner · November 10, 2021, 7:18am

It copies the data from the primary.

No. Nothing but Elasticsearch may change Elasticsearch's data. Quoting the same docs as last time:

WARNING : Don’t modify anything within the data directory or run processes that might interfere with its contents. If something other than Elasticsearch modifies the contents of the data directory, then Elasticsearch may fail, reporting corruption or other data inconsistencies, or may appear to work correctly having silently lost some of your data.

tropically · November 10, 2021, 12:39pm

Hi David
Querying from Master-node-0 I get 404 shards that are inconsistent.
As a result the health check is coming back red.

sh-4.2$ curl http://localhost:9200/_cluster/health
{"cluster_name":"elasticsearch","status":"red","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":789,"active_shards":1176,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":404,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":74.43037974683544}sh-4.2$

Would Elasticsearch still be able to resolve this to the replicas?

DavidTurner · November 10, 2021, 12:42pm

No, if the health is red then you have lost primaries too. You should close them and restore them from a recent snapshot.

tropically · November 10, 2021, 2:51pm

Hi David
The below output shows just one index that is in RED.
I would have to restore just that one correct? If we didn't have a snapshot then we should just delete that index?

sh-4.2$ wget -q 'http://localhost:9200/_cat/indices?v' -O- 
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   jaeger-service-2021-09-21 8QzFvB4dTjyt-W9RC2K-zg   5   1         37            1      146kb         93.3kb
yellow open   jaeger-service-2021-10-07 SbUPpDfdSwKQlK-9JcVNJw   5   1         78            2      161kb           99kb
green  open   jaeger-service-2021-11-04 5r1idiUcSfS-20JMppPgCA   5   1         59            2    230.6kb        115.3kb
yellow open   jaeger-span-2021-08-30    EA5KbnjQQZCQXqOtPtW5kA   5   1    3419896            0      157mb          157mb
yellow open   jaeger-span-2021-09-12    9zdLA7SvTDi7dZaEhVgcKA   5   1    3759025            0    278.6mb        174.3mb
yellow open   jaeger-span-2021-09-21    Rq1NSpFXSu6ZlHML7fTIew   5   1    3418116            0    253.9mb        158.5mb
yellow open   jaeger-service-2021-10-28 78GGztLBSXC7TB855OD_Tg   5   1         59            5    166.9kb        115.8kb
yellow open   jaeger-service-2021-10-09 EuUxVC99TfCLAmnKgqp96Q   5   1         33            1    141.5kb          103kb
yellow open   jaeger-service-2021-11-03 tUOy47QAR76-Xn1M8XoLOQ   5   1         70            4     97.3kb         97.3kb
yellow open   jaeger-service-2021-10-21 2CxwYPtrQCyrASXh3Gdviw   5   1         63            7    213.6kb        129.8kb
yellow open   jaeger-span-2021-10-14    WEH_nxnuRfyYC3SYAV4rIg   5   1    3781483            0    275.8mb        172.7mb
yellow open   jaeger-service-2021-10-15 gW8nICJiTei643Ex_EIGOw   5   1         36            0    185.1kb        112.9kb
yellow open   jaeger-span-2021-08-29    xDUGl0isRZefXgo0Li7JKQ   5   1    3419828            0    158.6mb        158.6mb
yellow open   jaeger-service-2021-10-12 OTfD1p1tQ2ae3g9xkoEkCw   5   1         55            5    195.5kb        121.9kb
yellow open   jaeger-service-2021-10-24 agIYV6FMRqyF7pFg9u-fQQ   5   1         32            0    167.4kb          109kb
yellow open   jaeger-span-2021-10-05    4UkwucmgTkyZDV3kZolo7g   5   1    2940953            0    221.5mb        138.2mb
yellow open   jaeger-service-2021-10-27 aIlNXXfnSP6_c8Wn5aW4Cw   5   1         57            7      215kb        141.3kb
yellow open   jaeger-span-2021-10-23    m5-G-pNIR_6BADGKCWA22Q   5   1    3518032            0    260.2mb        162.2mb
yellow open   jaeger-service-2021-09-27 QouFJ99VSEKzoy11uuGpNQ   5   1         36            0    196.3kb        118.3kb
yellow open   jaeger-span-2021-09-13    uN7Er1svQNCIBr1gNWABwA   5   1    3758504            0    274.4mb        171.3mb
yellow open   jaeger-service-2021-10-01 yEPyNcmHSrmmMRyekjv1QA   5   1         36            3    190.9kb        115.6kb
yellow open   jaeger-service-2021-09-14 3jnX2_iiTOS9zcKBlkpnHg   5   1         56            9    157.9kb        106.2kb
yellow open   jaeger-service-2021-11-01 bsJxj5CjTEefE_fFr48yTA   5   1         64            8    157.4kb        100.2kb
yellow open   jaeger-service-2021-10-04 oYxeWz-CRFiFWqATcNPwPg   5   1         35            0    186.8kb        111.9kb
yellow open   jaeger-span-2021-08-31    5WyEGqnNSY-_wmvmq8WF1g   5   1    3420536            0    158.4mb        158.4mb
yellow open   jaeger-span-2021-09-27    Uhnc9w39S9OhAsUZJQpNow   5   1    2827559            0    214.6mb        133.9mb
yellow open   jaeger-service-2021-09-24 x_ZBPUBsSg-_Ez48p_5YbQ   5   1         38            1    209.7kb          125kb
yellow open   jaeger-service-2021-10-19 -N0a1scwQWWjh4MfR1E60g   5   1         32            0    174.3kb          109kb
yellow open   jaeger-service-1970-01-01 3PzIv0_qSRmq_AxAWP_odA   5   1         53           15    103.4kb        103.4kb
green  open   jaeger-span-2021-11-10    QPGkCp9fQIqwb9m_MOwE9g   5   1    2593750            0    236.8mb        118.8mb
yellow open   jaeger-span-2021-10-22    XjtSvOUmSj2taq_P2D0aFQ   5   1    3509767            0    258.9mb        161.9mb
yellow open   jaeger-service-2021-09-17 fe4l-1JXTAOQXjSSb-JYZQ   5   1         63            7      190kb        122.6kb
yellow open   jaeger-span-2021-09-17    NHt56k4bSAmW_Za_vcNCwQ   5   1    3418743            0    252.9mb        157.9mb
yellow open   jaeger-span-2021-09-16    keT3dK5OQq6Ktvmj-oovMA   5   1    3586867            0    230.9mb        164.8mb
yellow open   jaeger-span-2021-09-11    frzB6LLmT927OeaSNDlaSg   5   1    3759080            0    278.8mb        173.9mb
yellow open   jaeger-service-2021-09-03 otsc3K16R1yLJK05aOvtAQ   5   1         32            0     79.7kb         79.7kb
yellow open   jaeger-span-2021-09-14    THoQzjk5SqijHScHgEKzxw   5   1    3758788            0    277.1mb        172.9mb
yellow open   jaeger-span-2021-09-19    tQzwpwAhRSqSRtvTWdvn7w   5   1    3417726            0    251.5mb          157mb
yellow open   jaeger-service-2021-10-10 AbSKRL0WRmenRv2ciyTUcA   5   1         33            0    178.1kb        109.4kb
yellow open   jaeger-span-2021-10-31    pJrE8S1BQFqB20F-vX3FJA   5   1    3815970            0    245.8mb        175.7mb
green  open   jaeger-span-2021-11-07    CYldGGsQTVivA0gg-rHm3A   5   1    4044528            0    365.9mb        182.9mb
yellow open   jaeger-service-2021-08-29 HNdvDMSLSVet68QWd9yZcA   5   1         31            0     72.9kb         72.9kb
yellow open   jaeger-span-2021-09-30    BOlg5ccxTzO9CRaz8-JbCQ   5   1    2827235            0    211.5mb        132.4mb
yellow open   jaeger-span-2021-09-18    yCe24PoDRAyWg9vzUOk2Nw   5   1    3418253            0      253mb        157.3mb
yellow open   jaeger-service-2021-09-26 1GeM_NjlQ4qIYcV31BeAEw   5   1         29            0    150.4kb         95.3kb
yellow open   jaeger-span-2021-10-11    xoQOiHoARVerBPmRX32o8w   5   1    3823711            0      277mb        173.4mb
yellow open   jaeger-span-2021-10-19    jhkj-ZtkR66z9mY2hV7oGw   5   1    3520879            0    260.9mb        163.1mb
green  open   jaeger-span-2021-11-04    Rj_zeHOeTHSSA08a_dfUow   5   1    4033151            0    366.1mb        182.7mb
yellow open   jaeger-span-2021-11-02    z7iTvsQUQTKchtI3ibrrcQ   5   1    3606838            0      168mb          168mb
yellow open   jaeger-service-2021-10-18 wwknDVwaSYyfyTx0NjLKdw   5   1         33            5    162.3kb          103kb
green  open   jaeger-service-2021-11-05 RaGDypprReuTGkz7VG2ABw   5   1         59            3    224.7kb        109.2kb
yellow open   jaeger-service-2021-11-02 LdiC782OQPKuHVXBlIbSLA   5   1         83           20    164.4kb        104.9kb
yellow open   jaeger-service-2021-08-31 gDS338OtTjyUFDabSL5hAA   5   1         55            0    123.6kb        123.6kb
yellow open   jaeger-service-2021-09-02 ANqXf9kGQrCHFWJamWPzpw   5   1         32            0     79.6kb         79.6kb
yellow open   jaeger-service-2021-10-08 KgFcWn_3RnmyN48pS_k1MQ   5   1         79           11    196.3kb        135.9kb
yellow open   jaeger-span-2021-09-10    C8ZyIW0sSsiHP8cKjJQp_w   5   1    3759812            0    277.4mb        173.8mb
yellow open   jaeger-span-2021-08-27    l_xzfGleSP68vz0QVDuakQ   5   1    2021778            0     94.4mb         94.4mb
yellow open   jaeger-service-2021-08-25 juNBB3mlSbe4Q2xcpXD6TQ   5   1         48            1     71.6kb         71.6kb
yellow open   jaeger-service-2021-08-27 1RbQII1JQsyvzEVvDrfBsw   5   1         34            4     76.8kb         76.8kb
yellow open   jaeger-span-2021-09-04    qDMyR9WkQ5G8P38x9cQ4pg   5   1    3758510            0      173mb          173mb
yellow open   jaeger-span-2021-09-06    6Y4zVAoPTU-wpwQaWAnrFg   5   1    3602934            0    167.4mb        167.4mb
yellow open   jaeger-span-1970-01-01    6LY4lKgZT1iJAbFA16hdhA   5   1      85404            0     18.9mb         18.9mb
yellow open   jaeger-span-2021-09-15    PVtBBX6vQZiddTJjPpipqA   5   1    3754867            0    278.1mb        173.8mb
green  open   jaeger-span-2021-11-05    AE2C4A79SdyABC83fH0Awg   5   1    4045842            0    367.2mb        183.6mb
yellow open   jaeger-service-2021-09-01 QUltamiLRBaxHDG1UXg2uw   5   1         31            0     76.1kb         76.1kb
green  open   jaeger-service-2021-11-09 vEYHh3waToqhTZ9bIGLseQ   5   1         78            4    209.9kb        108.1kb
yellow open   jaeger-span-2021-10-01    UhrUD8mDTI-wetf6WTGj6A   5   1    2783940            0    209.6mb        131.2mb
yellow open   jaeger-span-2021-11-01    M8THTlS8Tki_GNTl-eYWkw   5   1    3562843            0    263.7mb        164.6mb
yellow open   jaeger-service-2021-09-22 JBpswJb0R2228El-fepXhg   5   1         44            0    162.1kb        103.1kb
yellow open   jaeger-span-2021-10-07    zuxa9kEiS8CU2xf3uvRmEg   5   1    3527708            0      267mb        166.4mb
green  open   jaeger-span-2021-11-08    itNmxk1ORtqohk-0J0NHog   5   1    4045719            0    368.3mb        184.4mb
yellow open   jaeger-span-2021-10-04    hJfykD4SSLSI6zvWLSVCsQ   5   1    2828668            0    215.4mb        134.9mb
yellow open   jaeger-service-2021-10-14 0Dg97eKCTHGBpPH73Cblmw   5   1         36            2    172.6kb        106.7kb
yellow open   jaeger-span-2021-10-26    g-vukOV0QeKVhQ1lW4mxiw   5   1    3658642            0    268.9mb        167.7mb
yellow open   jaeger-span-2021-09-29    jo2IEqbJRAGllFfsD6BW2Q   5   1    2826751            0    213.4mb        132.9mb
yellow open   jaeger-span-2021-09-24    jwC-lsikRwCe2bN7dySlZA   5   1    2824679            0      212mb        132.6mb
yellow open   jaeger-span-2021-10-21    4EbKLYRRSvCam9UMCUN2Kw   5   1    3472462            0    251.8mb        157.9mb
yellow open   jaeger-span-2021-10-20    UvZgQWoiSMu62tMuPKMF3g   5   1    3527617            0    260.6mb        163.1mb
yellow open   jaeger-service-2021-10-25 4A_ZYcrnQUWqT1D1RGI_GQ   5   1         40            2    146.4kb        100.2kb
yellow open   jaeger-service-2021-10-17 cknO_22PTuqPu-_CihnPgw   5   1         32            2    152.3kb         96.5kb
yellow open   jaeger-service-2021-09-09 nC-4em23QQWU_skJd5Il3A   5   1         59            5    190.4kb        135.1kb
green  open   jaeger-service-2021-11-10 5yXuVW_xTgCbteii4Gnvyw   5   1         58            0    272.3kb        134.5kb
yellow open   jaeger-service-2021-09-29 x7Tqpt7VTfCBrNaullzaJQ   5   1         37            0      203kb        121.6kb
yellow open   jaeger-span-2021-10-16    8hpG9QW3QGW_bopwjtbgSw   5   1    3520005            0    257.2mb        160.9mb
yellow open   jaeger-service-2021-10-16 jAlaLt11Rbijuc-M0hc1RA   5   1         32            1      149kb         96.4kb
yellow open   jaeger-service-2021-10-26 HFa8dFhiRyu70yDbw2CaVg   5   1         36            4    126.9kb         90.5kb
yellow open   jaeger-span-2021-09-25    8davYgblThW145lcNQ11-g   5   1    2825632            0    213.6mb        133.5mb
yellow open   jaeger-service-2021-10-31 bQ8RwRMqSY6ACuki-ed1yg   5   1         33            1    155.3kb          103kb
yellow open   jaeger-span-2021-09-23    itIiT8GZTUSttz3jr9l-3A   5   1    3056171            0    229.4mb        143.5mb
yellow open   jaeger-service-2021-09-20 Nk7ztkUxTm-_CeD6LSyoQA   5   1         32            0    157.5kb         99.1kb
yellow open   jaeger-service-2021-08-30 G7BW345eT2CqYcypxxxL7A   5   1         32            0     82.7kb         82.7kb
yellow open   jaeger-span-2021-09-03    oRp6kd2OQ5G-5q6m840K-A   5   1    3754278            0    172.3mb        172.3mb
green  open   jaeger-span-2021-11-06    45IsjZsuS7WY7n-_eg__lg   5   1    4037166            0      366mb        183.1mb
yellow open   jaeger-span-2021-08-26    7G79rjUoTE62ww6CHFb5Bg   5   1    3388231            0    153.5mb        153.5mb
yellow open   jaeger-service-2021-10-13 vO5clTJhT-6ITMsKk2Iy4g   5   1         37            7    163.8kb          104kb
yellow open   jaeger-service-2021-09-10 F9hE3ym2So2tbJhxU4Ka7Q   5   1         57            1    200.4kb        124.5kb
yellow open   jaeger-span-2021-10-10    xx9zAY96TVSn5j_GJSuTxw   5   1    3859468            0    285.1mb        177.5mb
yellow open   jaeger-service-2021-09-13 BzlAJDBpTqqvngsfElXEhA   5   1         40            3    175.6kb        103.7kb
green  open   jaeger-service-2021-11-06 Jq1xiYkZRVGzgB09DDXTsA   5   1         34            0    206.1kb          103kb
yellow open   jaeger-span-2021-10-25    dG4iPa9wRce92n_QzGRsNg   5   1    3521222            0    261.2mb        162.7mb
red    open   jaeger-span-2021-11-03    4gd7PS_MQOGv_Y1YUWhWEg   5   1     858966            0     40.3mb         40.3mb
yellow open   jaeger-span-2021-10-03    rF8S1oarSmix_9PBQzOELQ   5   1    2828845            0    213.9mb        133.1mb
yellow open   jaeger-span-2021-09-08    t6MCQuUwSOK5lkBvpoBwJw   5   1    3412825            0    157.4mb        157.4mb
yellow open   jaeger-service-2021-10-06 8UXYbMU7R9ipn0fKqqnd-Q   5   1         79            6    131.6kb         89.3kb
sh-4.2$

DavidTurner · November 10, 2021, 4:13pm

Yes, just restore the red one, ES will recover the rest automatically.

tropically · November 10, 2021, 8:48pm

It seems we don't have the snapshot to recover from.
It is safe to delete it, understanding that we would lose that data.

DavidTurner · November 10, 2021, 9:11pm

Yes it won't cause Elasticsearch any problems to delete that index.

system · December 8, 2021, 9:12pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.