All shards failed for phase: [query] - messages after upgrade

Elastic Stack Elasticsearch
1

Hi,

I recently upgraded elasticsearch from 5.6.10 to 6.3.2 (on centos6). It's a single testing instance so no replication is configured yet. I dump telegraf metrics from various hosts in here.
Now after the upgrade when I start the service I see the following errors in the logs:
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_logstash_version_mismatch], reason [all shards failed]
[2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_cluster_status], reason [all shards failed]
[2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_version_mismatch], reason [all shards failed]
[2018-08-03T12:01:07,187][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_kibana_version_mismatch], reason [all shards failed]
[2018-08-03T12:01:07,258][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_version_mismatch]
[2018-08-03T12:01:07,258][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_logstash_version_mismatch]
[2018-08-03T12:01:07,258][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_kibana_version_mismatch]
[2018-08-03T12:01:07,259][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_cluster_status]

The node itself seems fine. Data is being ingested and can be read out. I just see these errors after service startup (there are more of them, but I just posted the top ones).
Some node details that might be helpful:
https://pastebin.com/fRPaYfiL

Can someone please explain what the above errors mean and how do I get rid of them?
If you need any more info, just ask.

Br,

2

Ok I figured it out. It was a watcher issue. The elasticsearch instance was a bit messy before the upgrade.
I listed the watches with:
https://www.elastic.co/guide/en/watcher/current/listing-watches.html
Then deleted them:
curl -X DELETE "localhost:9200/_watcher/watch/eXHJNE_jQ12dfLBwY6BXzw_logstash_version_mismatch"
... and so forth.
Errors are gone now.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.