Hi,
We are using ES5.4.1. In one of our production environment, a node left out of cluster and after it rejoined we are seeing master nodes giving these warning messages. Could anyone please take a look and clarify how it might affect the index/search operations ?
"Caused by: java.io.FileNotFoundException: no segments* file found in store(mmapfs(F:\data\nodes\0\indices\LnsZ8z0iQHCsfbnVc1OtxQ\1\index)):"
FullException -
[es-m03-rm] [sharedindex][1]: failed to list shard for shard_store on node [6xSE_CzOSXiig0Ehr-3r8A]
org.elasticsearch.action.FailedNodeException: Failed node [6xSE_CzOSXiig0Ehr-3r8A]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:246) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:160) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:218) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1050) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.lambda$handleException$17(TcpTransport.java:1451) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) [elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1449) [elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1441) [elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385) [elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.4.1.jar:5.4.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.transport.RemoteTransportException: [es-d06-rm][20.0.0.156:9300][internal:cluster/nodes/indices/shard/store[n]]
Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[sharedindex][1]]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:114) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:64) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:145) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:269) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:265) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1528) ~[elasticsearch-5.4.1-SNAPSHOT.jar:5.4.1-SNAPSHOT]
Caused by: java.io.FileNotFoundException: no segments* file found in store(mmapfs(F:\data\nodes\0\indices\LnsZ8z0iQHCsfbnVc1OtxQ\1\index)): files: [recovery.AV_YJEaSvIUUWZcP9zci._1fea.dii, recovery.AV_YJEaSvIUUWZcP9zci._1fea.dim, recovery.AV_YJEaSvIUUWZcP9zci._1fea.fdx, recovery.AV_YJEaSvIUUWZcP9zci._1fea.fnm, recovery.AV_YJEaSvIUUWZcP9zci._1fea.nvd,
...
Thanks,
Divya
Note: I had pasted the entire exception FullTrace earlier.