Hi,
Every now and again one shard of an index goes into initialization, no message in the log that indicate there is a problem.
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 10:59:56,301][INFO ][cluster.metadata ] [Bushwacker] [apache-varnish-2016.03.21] creating index, cause [auto(bulk api)], templates [wcms], shards [6]/[0], mappings [default, varnish.varnish
ncsa.log, varnish.varnishncsa.log]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 10:59:56,401][INFO ][cluster.metadata ] [Bushwacker] [apache-varnish-2016.03.21] update_mapping [varnish.varnishncsa.log]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 10:59:56,408][INFO ][cluster.metadata ] [Bushwacker] [apache-varnish-2016.03.21] update_mapping [varnish.varnishncsa.log]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 10:59:56,419][INFO ][cluster.metadata ] [Bushwacker] [apache-varnish-2016.03.21] update_mapping [varnish.varnishncsa.log]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 10:59:59,335][INFO ][cluster.metadata ] [Bushwacker] [apache-varnish-2016.03.21] update_mapping [varnish.varnishncsa.log]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 12:05:53,863][INFO ][cluster.metadata ] [Bushwacker] [apache-varnish-2016.03.21] update_mapping [varnish.varnishncsa.log]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 15:45:41,493][WARN ][index.engine ] [Bushwacker] [apache-varnish-2016.03.21][3] failed engine [refresh failed]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 15:45:41,493][WARN ][index.shard ] [Bushwacker] [apache-varnish-2016.03.21][3] Failed to perform scheduled engine refresh
/var/log/elasticsearch/elasticsearch.log:[apache-varnish-2016.03.21][[apache-varnish-2016.03.21][3]] RefreshFailedEngineException[Refresh failed]; nested: NullPointerException;
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 15:45:41,496][WARN ][indices.cluster ] [Bushwacker] [[apache-varnish-2016.03.21][3]] marking and sending shard failed due to [engine failure, reason [refresh failed]]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 15:45:41,497][WARN ][cluster.action.shard ] [Bushwacker] [apache-varnish-2016.03.21][3] received shard failed for [apache-varnish-2016.03.21][3], node[NPeu22XsRzK1cuFkjV7Ktw], [P], v[2], s[STARTED], a[id=
kNj3YPV4Q6OE7qOIET1t3w], indexUUID [r7HAjs1mQq2_kO4zyYFb7w], message [engine failure, reason [refresh failed]], failure [NullPointerException[null]]
/var/log/elasticsearch/elasticsearch.log:[2016-03-21 15:45:41,532][WARN ][index.translog ] [Bushwacker] [apache-varnish-2016.03.21][3] failed to delete temp file /opt/elasticsearch/es-data/elasticsearch/nodes/0/indices/apache-varnish-2016.03.21/3/tran
slog/translog-1522912244841259470.tlog
The shard changes every time but its only at a time,
After stopping logstash the index recovers, This has started happening after we split of logs indexes up from 1 index into 4?
I have slowed throughput rate of input (rabbitmq) and output(elasticsearch) with no lucky, there doesnt seem to be any memory problems.
Any ideas
Thanks
Wal