Accumulating watches for cluster status

I'm not too familiar with what exactly xpack sets up in terms of monitoring and how it interplays with the watcher api. But since the upgrade from 5.2.2 to 5.6 the install has degraded in performance and this excerpt from the logs seems like it might be an indicator why.

[2017-09-28T02:27:53,789][DEBUG][o.e.x.m.MonitoringService] [monitoring-elasticsearch-m-0] monitoring execution is skipped until previous execution terminated
[2017-09-28T02:28:01,115][DEBUG][o.e.x.w.e.ExecutionService] [monitoring-elasticsearch-m-0] executing watch [6wXowBzeSwufEbtr-bQJuQ_logstash_version_mismatch]
[2017-09-28T02:28:01,115][DEBUG][o.e.x.w.e.ExecutionService] [monitoring-elasticsearch-m-0] executing watch [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_cluster_status]
[2017-09-28T02:28:01,115][DEBUG][o.e.x.w.e.ExecutionService] [monitoring-elasticsearch-m-0] executing watch [6wXowBzeSwufEbtr-bQJuQ_kibana_version_mismatch]
[2017-09-28T02:28:01,115][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_logstash_version_mismatch_c51296ca-46a0-409b-8685-0f4ede82fc9f-2017-09-28T02:27:25.443Z] found [0] hits
[2017-09-28T02:28:01,116][DEBUG][o.e.x.w.e.ExecutionService] [monitoring-elasticsearch-m-0] executing watch [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_version_mismatch]
[2017-09-28T02:28:01,117][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_logstash_version_mismatch_c51296ca-46a0-409b-8685-0f4ede82fc9f-2017-09-28T02:27:25.443Z] found [0] hits
[2017-09-28T02:28:01,119][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_kibana_version_mismatch_62b827b7-cdf3-46d0-8aa3-5a0ca92b934e-2017-09-28T02:27:25.443Z] found [12] hits
[2017-09-28T02:28:01,120][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_kibana_version_mismatch_62b827b7-cdf3-46d0-8aa3-5a0ca92b934e-2017-09-28T02:27:25.443Z] found [0] hits
[2017-09-28T02:28:01,122][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_version_mismatch_3eb235d7-4168-4820-bca4-5ca456f6cd56-2017-09-28T02:27:25.443Z] found [0] hits
[2017-09-28T02:28:01,122][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_cluster_status_0363bd80-6f9f-49e0-86b8-8a7384e43e2b-2017-09-28T02:27:25.443Z] found [73047] hits
[2017-09-28T02:28:01,122][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_cluster_status_0363bd80-6f9f-49e0-86b8-8a7384e43e2b-2017-09-28T02:27:25.443Z] hit [{
  "error" : "Can not start an object, expecting field name (context: Object)"
}]
[2017-09-28T02:28:01,122][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_version_mismatch_3eb235d7-4168-4820-bca4-5ca456f6cd56-2017-09-28T02:27:25.443Z] found [0] hits
[2017-09-28T02:28:01,123][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_cluster_status_0363bd80-6f9f-49e0-86b8-8a7384e43e2b-2017-09-28T02:27:25.443Z] found [1] hits
[2017-09-28T02:28:01,123][DEBUG][o.e.x.w.i.s.ExecutableSimpleInput] [monitoring-elasticsearch-m-0] [6wXowBzeSwufEbtr-bQJuQ_elasticsearch_cluster_status_0363bd80-6f9f-49e0-86b8-8a7384e43e2b-2017-09-28T02:27:25.443Z] hit [{
  "error" : "Can not start an object, expecting field name (context: Object)"
}]

It seems to me that there is an error in the watch setup to monitor the cluster status and it is forever accumulating hits. Since I am on the basic license as of a few days ago with the trial running out I can not manipulate or look further into the watcher specifics. Is there some guidance on how to reset the watcher? I believe this is related to the growing gaps in monitoring graphs for the 10s intervals.

I've stood up a 2nd elasticsearch cluster and see this same behavior before indexing anything. The number of hits for the cluster_status keeps growing and it has the same error. This is before any loading of templates.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.