A bunch of exceptions during node startup in a single node cluster

Elastic Stack Elasticsearch
1

Hello,

During every startup on my elasticsearch instance I receive a bunch of exceptions and I'm not sure if it's okey...

I'm using elasticsearch 5.x with x-pack (basic license). Just updated to 5.3.0 version. My setup consists of a single node cluster.

The first exception has type "org.elasticsearch.cluster.block.ClusterBlockException" and repeats couple of times:

[2017-03-29T10:22:57,849][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [my_node_name] collector [index-recovery-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.3.0.jar:5.3.0]

The second exception has type "org.elasticsearch.xpack.monitoring.exporter.ExportException" :

[2017-03-29T10:23:04,811][INFO ][o.e.l.LicenseService     ] [my_node_name] license [my_key] mode [basic] - valid
[2017-03-29T10:23:04,823][INFO ][o.e.g.GatewayService     ] [my_node_name] recovered [155] indices into cluster_state
[2017-03-29T10:25:10,266][WARN ][o.e.x.m.MonitoringService] [my_node_name] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.<init>(ExportBulk.java:106) ~[?:?]

I can't post full tracebacks in "code" tags for both exceptions, because of a length limit for posts.

I want to know if it's ok that these exceptions are thrown and if not, what should I do?

2 
[2017-03-29T10:22:57,849][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [my_node_name] collector [index-recovery-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.3.0.jar:5.3.0]

This exception is the monitoring plugin trying to collect data, however, the cluster is not yet available as the cluster state has not been recovered. It's safe to ignore as long as the cluster state is eventually recovered (I assume it does once the nodes have finished starting up)

[2017-03-29T10:23:04,811][INFO ][o.e.l.LicenseService     ] [my_node_name] license [my_key] mode [basic] - valid
[2017-03-29T10:23:04,823][INFO ][o.e.g.GatewayService     ] [my_node_name] recovered [155] indices into cluster_state
[2017-03-29T10:25:10,266][WARN ][o.e.x.m.MonitoringService] [my_node_name] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.<init>(ExportBulk.java:106) ~[?:?]

It's hard to tell what the cause of this one is, it looks like it's failed to export monitoring data, can you post the full stack trace for this exception somewhere and link to it here?

3

Thanks for the reply, dakrone!

Yes, the first exception repeats couple of times, then it disappears.

The full traceback for the second exception: traceback

4

Okay, looking at your log message - are you saving the monitoring data into the same cluster or sending it to a remote cluster? (I am guessing the same cluster). It looks like your cluster is red, so the bulk ingestion is failing.

Does the problem persist once the cluster recovers to at least yellow?

/cc @pickypg did I miss anything important about this?

5

I'm saving monitoring data into the same cluster.

Nope, all listed exceptions disappear after the message:

[2017-03-29T10:25:33,618][INFO ][o.e.c.r.a.AllocationService] [my_node_name] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-data-2][0], [logstash-2017.01.25][0], [logstash-2017.01.26][0]] ...]).

So everything is working as it should and I have no reasons to worry, right?

6

Ivan writes:

So everything is working as it should and I have no reasons to worry,
right?

That's correct.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.