at java.lang.Thread.run(Thread.java:835) ~[?:?]
at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) ~[?:?]
Caused by: java.io.IOException: Input/output error: NIOFSIndexInput(path="/data/elasticsearch/nodes/0/indices/fIQlaITkRGeU73pEZ6upjw/0/index/_b2c.fdt")
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:189) ~[lucene-core-8.1.0.jar:8.1.0 dbe5ed0b2f17677ca6c904ebae919363f2d36a0a - ishan - 2019-05-09 19:34:03]
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:160) ~[lucene-core-8.1.0.jar:8.1.0 dbe5ed0b2f17677ca6c904ebae919363f2d36a0a - ishan - 2019-05-09 19:34:03]
at org.elasticsearch.common.lucene.store.InputStreamIndexInput.read(InputStreamIndexInput.java:73) ~[elasticsearch-7.3.1.jar:7.3.1]
I'm seeing exception in the destination recovery node.
That shard has been in "relocating" state for more than a day. The exception is quiet large so I copy & pasted the part I think might be helpful...
And this is a primary shard so I can't do the trick to set replica_count to 0 then back to 1.
That directory from above log has many files but doesn't contain the file _b2c.fdt.
What does this mean? How do I recover it?