After updating to ES 5.1.1 and upgrading the machine OS to CentOS 7.2, I am seeing the cluster turn to RED, with the most recent logstash index having unassigned shards.
[ageorges@tangela2 ~]$ curl -XGET '192.168.24.1:9200/_cat/health'
1481815088 16:18:08 tangela red 2 2 921 476 0 2 31 0 - 96.5%
[ageorges@tangela2 ~]$ curl -XGET '192.168.24.1:9200/_cat/shards?pretty' 2>/dev/null | grep -v STARTED
logstash-2016.12.15 4 p UNASSIGNED
logstash-2016.12.15 4 r UNASSIGNED
The file referred to below in the trace lines
[2016-12-15T08:03:40,589][INFO ][o.e.c.r.a.AllocationService] [tangela1.ugent.be] Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[logstash-2016.12.15][4]] ...]).
[2016-12-15T13:36:44,191][WARN ][o.e.c.a.s.ShardStateAction] [tangela1.ugent.be] [logstash-2016.12.15][3] received shard failed for shard id [[logstash-2016.12.15][3]], allocation id [JOxwRSAYQtG-5wEMONJkNA], primary term [0], message [shard failure, reason [refresh failed]], failure [CorruptIndexException[compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/AtvVrqs_RP6Asrc7d14T_w/3/index/_1oq.dii")))]]
org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/AtvVrqs_RP6Asrc7d14T_w/3/index/_1oq.dii")))
And
[root@tangela1 ~]# ls -l /var/lib/elasticsearch/tangela/nodes/0/indices/AtvVrqs_RP6Asrc7d14T_w/3/index/_1oq.dii
ls: cannot access /var/lib/elasticsearch/tangela/nodes/0/indices/AtvVrqs_RP6Asrc7d14T_w/3/index/_1oq.dii: No such file or directory
For the logs, see https://gist.github.com/itkovian/ead08bbd229ac6509cdd5cc82220a046
-- Andy