I have a 2 node ES cluster (2.3.3 on ubuntu) which shows green, and I am getting the below spamming my logfiles 6.8GB daily. The date of the shard varies between 2015 and 2016.
I have checked _cat/shards/ and the referenced shard logstash-2016.01.19 does not appear, nor does any reference to logstash at all.
I have checked _cat/indices/ and all of the indices show up as green, and again no reference to logstash in at all.
On the filesystem in /var/lib/elasticsearch/Clustername/nodes/ the 1 folder does not exist on either of the nodes.
Any help finding out why Elasticsearch thinks there are shards that don't exist would be greatly appreciated.
[2016-07-04 17:35:03,317][DEBUG][action.admin.indices.stats] [server] [indices:monitor/stats] failed to execute operation for shard [[logstash-2016.01.19][3], node[2fEASGQ-SGaNeflA0WoTeA], [P], v[14], s[STARTED], a[id=3JuxCZj3SKqu7Efo26VaxQ]]
ElasticsearchException[failed to refresh store stats]; nested: NoSuchFileException[/var/lib/elasticsearch/Clustername/nodes/1/indices/logstash-2016.01.19/3/index];
at org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1534)
at org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1519)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.index.store.Store.stats(Store.java:293)
at org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:665)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:134)
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:165)
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47)
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:409)
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:388)
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:375)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException: /var/lib/elasticsearch/Clustername/nodes/1/indices/logstash-2016.01.19/3/index
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:190)
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:202)
at org.elasticsearch.index.store.FsDirectoryService$1.listAll(FsDirectoryService.java:127)
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)
at org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1540)
at org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1532)
... 15 more