We keep getting the NoSuchFileException on seemly random snowstorm_metrics-* indexes. Closing the offending index and re-opening it corrects the problem for a little while. Then the exception appears again usually on a different index.
The offending index is green and open but the docs.count, docs.deleted, store.size and pri.store.size are all blank. Similar to closed indexes.
The offending shards are started but several columns are blank. This only seems to happen on the offending shards all the others are populated with values.
Following is a log file snippet:
[2017-01-25 20:15:25,056][DEBUG][action.admin.indices.stats] [c3pupclog2] [indices:monitor/stats] failed to execute operation for shard [[snowstorm_metrics-2017.01.10][0], node[m7W7M4eLRT2GtDjA0QgsuQ], [P], v[3], s[STARTED], a[id=AoVkTo8wRXyILXdbonS_LA]]
ElasticsearchException[failed to refresh store stats]; nested: NoSuchFileException[/opt/app/elasticsearch/data/snowstorm_logs_prod/nodes/0/indices/snowstorm_metrics-2017.01.10/0/index];
at org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1532)
This stack is used for ELK - elastic version is 2.4.1