Elasticsearch errors

Elastic Stack Elasticsearch
1

I ..I have 3 node of cluster . I have installed x pack . Everything was working fine but by mistake , one node of my cluster got disconnected . So I restarted cluster .. now it is giving these errors .

[2017-06-16T18:13:35,340][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [slave-1] collector [indices-stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

After few exceptions -

[2017-06-16T18:13:35,344][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [slave-1] collector [index-recovery] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

then

[2017-06-16T18:13:35,350][ERROR][o.e.x.m.c.i.IndexStatsCollector] [slave-1] collector [index-stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

and

[2017-06-16T18:14:03,422][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_cluster_status]
java.lang.IllegalStateException: watch store not started

then

[2017-06-16T18:14:03,426][ERROR][o.e.x.w.t.a.g.TransportGetWatchAction] [slave-1] failed to get watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_version_mismatch]
java.lang.IllegalStateException: watch store not started

then

[2017-06-16T18:14:03,429][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_version_mismatch]
java.lang.IllegalStateException: watch store not started

and so on. I had deleted the previous index before and then restarted it .. Since then , it is giving this .. even though cluster is working .. only monitoring section is not accessible .

Can anyone help me with this ?

2

When you say "Monitoring section is not accessible" -- what do you get when you try to access Monitoring?

Can you share the output of a search in some Monitoring data?

GET /.monitoring-data-2/cluster_info/_search?filter_path=hits.hits._source.cluster_uuid,hits.hits._source.license.status
3

yeah .. sometimes it works fine , sometimes it shows this ->

4

Can you check that your clusters have valid licenses?

I edited the query that I pasted above so that I'm not asking you to share private info - the query just gets cluster UUID and license status from the monitoring data. Can you run that query so we can check if this is a license issue?

5

Sorry for the late reply .
Btw It is showing license is active .

6

Do you have any templates that might have patterns which clobber the monitoring data?

Try the http://localhost:9200/_template/ API and look for templates with index patterns that possibly match match monitoring indices.

7

Hi..
I tried http://localhost:9200/_template/
Instead of localhost , I had to use IP of the node and I got this;

Screenshot from 2017-06-28 15-12-53.png1847×954 390 KB

8

Have you added any of your own templates to Elasticsearch?

If so, should check that you haven't created a template with an index pattern that might match the .monitoring* indices.

You should paste your template output into a JSON formatter and look at each template's index pattern to make sure it wouldn't mistakenly match the .monitoring* indices.

9

Hey .. sorry for the late response .

I haven't created my own template , I am going on with the default configuration till now .
I do want to mention that the kibana monitoring section some times works , sometimes doesn't . Right now , it's working .
In fact, I haven't faced this error for 3-4 days . I have changed nothing , I don't know why there's no error.
As far as , I remember , when I was having this error , perhaps , there was some logs in elasticsearch console , indicating , failed to start monitoring shards or something like this.
Unfortunately , I couldn't take screenshot , and I have also deleted the old logs in elasticsaerch log folder
Next time , if it occurs , I'll share the screenshot with you in PM.

Thanks

10

Can I make a hypothesis and guess that when the error happens, the time picker date range was expanded to include a time that the cluster had a different license type?

I'll continue to investigate. Stay tuned.

11

No, I don't think so , I always use last 2 years in time picker . If it is a license issue then, it should be behaving monotonically either it would work or it won't . No combination of both cases can happen .

12

I was just doing a POC and noticed on my elastic search today that i was getting a similar error. I think mine has to do with license issue. i ran the below with the results posted below:
GET /.monitoring-data-2/cluster_info/_search?filter_path=hits.hits._source.cluster_uuid,hits.hits._source.license.status
result:
{
"hits": {
"hits": [
{
"_source": {
"cluster_uuid": "ZaAaQ17FQbuQtfzIxFpO6g",
"license": {
"status": "active"
}
}
}
]
}
}

Log stack :

[ERROR][o.e.x.w.t.a.g.TransportGetWatchAction] [8NX03Oq] failed to get watch [ZaAaQ17FQbuQtfzIxFpO6g_elasticsearch_cluster_status]
java.lang.IllegalStateException: watch store not started
at org.elasticsearch.xpack.watcher.watch.WatchStore.ensureStarted(WatchStore.java:367) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchStore.get(WatchStore.java:153) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.getWatch(WatcherService.java:158) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.get.TransportGetWatchAction.masterOperation(TransportGetWatchAction.java:86) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.get.TransportGetWatchAction.masterOperation(TransportGetWatchAction.java:53) ~[?:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:163) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:126) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:104) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.transport.actions.WatcherTransportAction.doExecute(WatcherTransportAction.java:56) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.WatcherTransportAction.doExecute(WatcherTransportAction.java:40) ~[?:?]

13

I am running into the same situation.

14

DId you get to resolve this ? I get the same problem. I am actually trying to install both logstash and elasticsearch. Elastic search logs fail with

[2017-08-03T21:42:02,543][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [aW-uiux4TDm7PODVVN6ZIg_logstash_version_mismatch]
java.lang.IllegalStateException: watch store not started
at org.elasticsearch.xpack.watcher.watch.WatchStore.ensureStarted(WatchStore.java:367) ~[?:?]

15

It seems that the failed to get watch errors and watch store not started can be safely ignored. What that means is that the Monitoring plugin for Elasticsearch is trying to add watches, but Elasticsearch isn't initialized far enough for that to work. The Monitoring plugin will keep trying and adding the watch should work the next time.

If you are concerned that Watcher is not working in the cluster or that the Monitoring plugin is getting stuck trying to send the watches, you could try manually executing the watches by ID. The ID of the watches is in the error message. So if you see:

[2017-06-16T18:14:03,422][ERROR][o.e.x.m.e.l.LocalExporter] failed to get monitoring watch [8ix4VdBmSJOml0PwEdjt8w_elasticsearch_cluster_status]
java.lang.IllegalStateException: watch store not started

You could try running:

POST /_xpack/watcher/watch/8ix4VdBmSJOml0PwEdjt8w_elasticsearch_cluster_status/_execute

To execute the watch manually and make sure that it is working.

But the bottom line is that these error messages are temporary and can be ignored. It could be happening when the cluster is in recovery due to a node rejoining the cluster. Monitoring will keep trying to do its job to add the watches to the cluster, but it'll fail because the cluster isn't ready. The watches will be added after the cluster recovers because the Monitoring plugin will keep re-trying.

16

Thanks!

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.