I ..I have 3 node of cluster . I have installed x pack . Everything was working fine but by mistake , one node of my cluster got disconnected . So I restarted cluster .. now it is giving these errors .
[2017-06-16T18:13:35,340][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [slave-1] collector [indices-stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
After few exceptions -
[2017-06-16T18:13:35,344][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [slave-1] collector [index-recovery] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
then
[2017-06-16T18:13:35,350][ERROR][o.e.x.m.c.i.IndexStatsCollector] [slave-1] collector [index-stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
and
[2017-06-16T18:14:03,422][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_cluster_status]
java.lang.IllegalStateException: watch store not started
then
[2017-06-16T18:14:03,426][ERROR][o.e.x.w.t.a.g.TransportGetWatchAction] [slave-1] failed to get watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_version_mismatch]
java.lang.IllegalStateException: watch store not started
then
[2017-06-16T18:14:03,429][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_version_mismatch]
java.lang.IllegalStateException: watch store not started
and so on. I had deleted the previous index before and then restarted it .. Since then , it is giving this .. even though cluster is working .. only monitoring section is not accessible .
Can you check that your clusters have valid licenses?
I edited the query that I pasted above so that I'm not asking you to share private info - the query just gets cluster UUID and license status from the monitoring data. Can you run that query so we can check if this is a license issue?
Have you added any of your own templates to Elasticsearch?
If so, should check that you haven't created a template with an index pattern that might match the .monitoring* indices.
You should paste your template output into a JSON formatter and look at each template's index pattern to make sure it wouldn't mistakenly match the .monitoring* indices.
I haven't created my own template , I am going on with the default configuration till now .
I do want to mention that the kibana monitoring section some times works , sometimes doesn't . Right now , it's working .
In fact, I haven't faced this error for 3-4 days . I have changed nothing , I don't know why there's no error.
As far as , I remember , when I was having this error , perhaps , there was some logs in elasticsearch console , indicating , failed to start monitoring shards or something like this.
Unfortunately , I couldn't take screenshot , and I have also deleted the old logs in elasticsaerch log folder
Next time , if it occurs , I'll share the screenshot with you in PM.
Can I make a hypothesis and guess that when the error happens, the time picker date range was expanded to include a time that the cluster had a different license type?
No, I don't think so , I always use last 2 years in time picker . If it is a license issue then, it should be behaving monotonically either it would work or it won't . No combination of both cases can happen .
I was just doing a POC and noticed on my elastic search today that i was getting a similar error. I think mine has to do with license issue. i ran the below with the results posted below:
GET /.monitoring-data-2/cluster_info/_search?filter_path=hits.hits._source.cluster_uuid,hits.hits._source.license.status
result:
{
"hits": {
"hits": [
{
"_source": {
"cluster_uuid": "ZaAaQ17FQbuQtfzIxFpO6g",
"license": {
"status": "active"
}
}
}
]
}
}
Log stack :
[ERROR][o.e.x.w.t.a.g.TransportGetWatchAction] [8NX03Oq] failed to get watch [ZaAaQ17FQbuQtfzIxFpO6g_elasticsearch_cluster_status]
java.lang.IllegalStateException: watch store not started
at org.elasticsearch.xpack.watcher.watch.WatchStore.ensureStarted(WatchStore.java:367) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchStore.get(WatchStore.java:153) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.getWatch(WatcherService.java:158) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.get.TransportGetWatchAction.masterOperation(TransportGetWatchAction.java:86) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.get.TransportGetWatchAction.masterOperation(TransportGetWatchAction.java:53) ~[?:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:163) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:126) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:104) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.transport.actions.WatcherTransportAction.doExecute(WatcherTransportAction.java:56) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.WatcherTransportAction.doExecute(WatcherTransportAction.java:40) ~[?:?]
DId you get to resolve this ? I get the same problem. I am actually trying to install both logstash and elasticsearch. Elastic search logs fail with
[2017-08-03T21:42:02,543][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [aW-uiux4TDm7PODVVN6ZIg_logstash_version_mismatch]
java.lang.IllegalStateException: watch store not started
at org.elasticsearch.xpack.watcher.watch.WatchStore.ensureStarted(WatchStore.java:367) ~[?:?]
It seems that the failed to get watch errors and watch store not started can be safely ignored. What that means is that the Monitoring plugin for Elasticsearch is trying to add watches, but Elasticsearch isn't initialized far enough for that to work. The Monitoring plugin will keep trying and adding the watch should work the next time.
If you are concerned that Watcher is not working in the cluster or that the Monitoring plugin is getting stuck trying to send the watches, you could try manually executing the watches by ID. The ID of the watches is in the error message. So if you see:
[2017-06-16T18:14:03,422][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_cluster_status]
java.lang.IllegalStateException: watch store not started
You could try running:
POST /_xpack/watcher/watch/8ix4VdBmSJOml0PwEdjt8w_elasticsearch_cluster_status/_execute
To execute the watch manually and make sure that it is working.
But the bottom line is that these error messages are temporary and can be ignored. It could be happening when the cluster is in recovery due to a node rejoining the cluster. Monitoring will keep trying to do its job to add the watches to the cluster, but it'll fail because the cluster isn't ready. The watches will be added after the cluster recovers because the Monitoring plugin will keep re-trying.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
