Hi,
I recently upgraded elasticsearch from 5.6.10 to 6.3.2 (on centos6). It's a single testing instance so no replication is configured yet. I dump telegraf metrics from various hosts in here.
Now after the upgrade when I start the service I see the following errors in the logs:
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,185][DEBUG][o.e.a.s.TransportSearchAction] [node.mydomain.com] All shards failed for phase: [query]
[2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_logstash_version_mismatch], reason [all shards failed]
[2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_cluster_status], reason [all shards failed]
[2018-08-03T12:01:07,186][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_version_mismatch], reason [all shards failed]
[2018-08-03T12:01:07,187][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node.mydomain.com] failed to execute [search] input for watch [eXHJNE_jQ12dfLBwY6BXzw_kibana_version_mismatch], reason [all shards failed]
[2018-08-03T12:01:07,258][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_version_mismatch]
[2018-08-03T12:01:07,258][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_logstash_version_mismatch]
[2018-08-03T12:01:07,258][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_kibana_version_mismatch]
[2018-08-03T12:01:07,259][WARN ][o.e.x.w.e.ExecutionService] [node.mydomain.com] failed to execute watch [eXHJNE_jQ12dfLBwY6BXzw_elasticsearch_cluster_status]
The node itself seems fine. Data is being ingested and can be read out. I just see these errors after service startup (there are more of them, but I just posted the top ones).
Some node details that might be helpful:
https://pastebin.com/fRPaYfiL
Can someone please explain what the above errors mean and how do I get rid of them?
If you need any more info, just ask.
Br,