# Service shutdown with unexpected error while indexing monitoring document

**URL:** <https://discuss.elastic.co/t/service-shutdown-with-unexpected-error-while-indexing-monitoring-document/138005>\
**Category:** Elasticsearch\
**Created:** [June 29, 2018, 4:25pm UTC](https://discuss.elastic.co/t/service-shutdown-with-unexpected-error-while-indexing-monitoring-document/138005 "2018-06-29T16:25:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gthang](https://avatars.discourse-cdn.com/v4/letter/g/258eb7/32.png) [@gthang](https://discuss.elastic.co/u/gthang)\
**Post date:** [June 29, 2018, 4:25pm UTC](https://discuss.elastic.co/t/service-shutdown-with-unexpected-error-while-indexing-monitoring-document/138005/1 "2018-06-29T16:25:41Z")

</div>

We upgraded our instances to 6.3 last week, and today our elastic instances took a dive. Here are the output from the logs on 1 of the nodes. We have disabled monitoring and restarted and things are starting to collect again.

Any ideas on what could be off? Please let me know if you would like any more debug information?

```
[2018-06-29T10:35:35,542][WARN][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[stats-dny4i1stc02-5][10.6.3.42:9008][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[stats-dny4i1stc02-5][10.6.3.42:9008][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$7@2b51d373 on EsThreadPoolExecutor[name = stats-dny4i1stc02-5/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@3bb2232e[Running, pool size = 36, active threads = 36, queued tasks = 200, completed tasks = 14140094]]];
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_66]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_66]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_66]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_66]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_66]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_66]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_66]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_66]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_66]
```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 3, 2018, 8:22am UTC](https://discuss.elastic.co/t/service-shutdown-with-unexpected-error-while-indexing-monitoring-document/138005/2 "2018-07-03T08:22:30Z")

</div>

This exception is returned from the elasticsearch cluster you are trying to indexing the monitoring data into (might be local or remote). The error message hints that the cluster is busy and overloaded. Is that cluster running over capacity?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 31, 2018, 8:22am UTC](https://discuss.elastic.co/t/service-shutdown-with-unexpected-error-while-indexing-monitoring-document/138005/3 "2018-07-31T08:22:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.