# All shards failed for phase: \[query\] - messages after upgrade **URL:** \ **Category:** Elasticsearch\ **Created:** [August 3, 2018, 11:56am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-messages-after-upgrade/142900 "2018-08-03T11:56:26Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Pimpek](https://avatars.discourse-cdn.com/v4/letter/p/49beb7/32.png) [@Pimpek](https://discuss.elastic.co/u/Pimpek)\ **Post date:** [August 3, 2018, 11:56am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-messages-after-upgrade/142900/1 "2018-08-03T11:56:27Z")
Hi, I recently upgraded elasticsearch from 5.6.10 to 6.3.2 (on centos6). It's a single testing instance so no replication is configured yet. I dump telegraf metrics from various hosts in here. Now after the upgrade when I start the service I see the following errors in the logs: [2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [[node.mydomain.com](http://node.mydomain.com)] All shards failed for phase: [query] [2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [[node.mydomain.com](http://node.mydomain.com)] All shards failed for phase: [query] [2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [[node.mydomain.com](http://node.mydomain.com)] All shards failed for phase: [query] [2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [[node.mydomain.com](http://node.mydomain.com)] All shards failed for phase: [query] [2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [[node.mydomain.com](http://node.mydomain.com)] failed to execute [search] input for watch [eXHJNE\_jQ12dfLBwY6BXzw\_logstash\_version\_mismatch], reason [all shards failed] [2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [[node.mydomain.com](http://node.mydomain.com)] failed to execute [search] input for watch [eXHJNE\_jQ12dfLBwY6BXzw\_elasticsearch\_cluster\_status], reason [all shards failed] [2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [[node.mydomain.com](http://node.mydomain.com)] failed to execute [search] input for watch [eXHJNE\_jQ12dfLBwY6BXzw\_elasticsearch\_version\_mismatch], reason [all shards failed] [2018-08-03T12:01:07,187][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [[node.mydomain.com](http://node.mydomain.com)] failed to execute [search] input for watch [eXHJNE\_jQ12dfLBwY6BXzw\_kibana\_version\_mismatch], reason [all shards failed] [2018-08-03T12:01:07,258][WARN][o.e.x.w.e.ExecutionService] [[node.mydomain.com](http://node.mydomain.com)] failed to execute watch [eXHJNE\_jQ12dfLBwY6BXzw\_elasticsearch\_version\_mismatch] [2018-08-03T12:01:07,258][WARN][o.e.x.w.e.ExecutionService] [[node.mydomain.com](http://node.mydomain.com)] failed to execute watch [eXHJNE\_jQ12dfLBwY6BXzw\_logstash\_version\_mismatch] [2018-08-03T12:01:07,258][WARN][o.e.x.w.e.ExecutionService] [[node.mydomain.com](http://node.mydomain.com)] failed to execute watch [eXHJNE\_jQ12dfLBwY6BXzw\_kibana\_version\_mismatch] [2018-08-03T12:01:07,259][WARN][o.e.x.w.e.ExecutionService] [[node.mydomain.com](http://node.mydomain.com)] failed to execute watch [eXHJNE\_jQ12dfLBwY6BXzw\_elasticsearch\_cluster\_status] The node itself seems fine. Data is being ingested and can be read out. I just see these errors after service startup (there are more of them, but I just posted the top ones). Some node details that might be helpful: [https://pastebin.com/fRPaYfiL](https://pastebin.com/fRPaYfiL) Can someone please explain what the above errors mean and how do I get rid of them? If you need any more info, just ask. Br, ---
**Author:** ![Pimpek](https://avatars.discourse-cdn.com/v4/letter/p/49beb7/32.png) [@Pimpek](https://discuss.elastic.co/u/Pimpek)\ **Post date:** [August 3, 2018, 12:46pm UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-messages-after-upgrade/142900/2 "2018-08-03T12:46:29Z")
Ok I figured it out. It was a watcher issue. The elasticsearch instance was a bit messy before the upgrade. I listed the watches with: [https://www.elastic.co/guide/en/watcher/current/listing-watches.html](https://www.elastic.co/guide/en/watcher/current/listing-watches.html) Then deleted them: `curl -X DELETE "localhost:9200/_watcher/watch/eXHJNE_jQ12dfLBwY6BXzw_logstash_version_mismatch"` ... and so forth. Errors are gone now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 31, 2018, 12:46pm UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-messages-after-upgrade/142900/3 "2018-08-31T12:46:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.