Service shutdown with unexpected error while indexing monitoring document

Elastic Stack Elasticsearch
1

We upgraded our instances to 6.3 last week, and today our elastic instances took a dive. Here are the output from the logs on 1 of the nodes. We have disabled monitoring and restarted and things are starting to collect again.

Any ideas on what could be off? Please let me know if you would like any more debug information?

[2018-06-29T10:35:35,542][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[stats-dny4i1stc02-5][10.6.3.42:9008][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[stats-dny4i1stc02-5][10.6.3.42:9008][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$7@2b51d373 on EsThreadPoolExecutor[name = stats-dny4i1stc02-5/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@3bb2232e[Running, pool size = 36, active threads = 36, queued tasks = 200, completed tasks = 14140094]]];
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_66]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_66]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_66]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_66]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_66]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_66]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_66]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_66]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_66]
2

This exception is returned from the elasticsearch cluster you are trying to indexing the monitoring data into (might be local or remote). The error message hints that the cluster is busy and overloaded. Is that cluster running over capacity?

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.