# ElasticSearch Monitoring Failing

**URL:** <https://discuss.elastic.co/t/elasticsearch-monitoring-failing/138435>\
**Category:** Elasticsearch\
**Created:** [July 3, 2018, 5:48pm UTC](https://discuss.elastic.co/t/elasticsearch-monitoring-failing/138435 "2018-07-03T17:48:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tebriel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tebriel/32/44889_2.png) [@tebriel](https://discuss.elastic.co/u/tebriel)\
**Post date:** [July 3, 2018, 5:48pm UTC](https://discuss.elastic.co/t/elasticsearch-monitoring-failing/138435/1 "2018-07-03T17:48:41Z")

</div>

Upgraded to 6.3.0, elasticsearch monitoring now fails, stack trace follows. I'm seeing occasional bursts of data showing up in Kibana 6.3.0, but it's got gaps in data. I tried increasing shards to 3 (from the 1 in the template) and deleting the index, no improvement.  
Monitoring is enabled via the persistent settings API:

```auto
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

```

```auto
[2018-07-03T17:43:08,357][WARN][o.e.x.m.MonitoringService] [m0_nP62] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.<init>(ExportBulk.java:95) ~[x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1.onFailure(ExportBulk.java:93) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:206) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:200) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:96) [x-pack-core-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:164) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:68) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:135) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:111) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:570) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:379) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:374) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:897) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:855) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.lambda$handleException$34(TcpTransport.java:1510) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1508) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1500) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1430) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) [transport-netty4-6.3.0.jar:6.3.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
<snip to be under max char count>
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
	at java.lang.Thread.run(Thread.java:844) [?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:156) [x-pack-monitoring-6.3.0.jar:6.3.0]
	... 46 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:124) ~[?:?]
	... 44 more

```

---

<div class="post-metadata">

**Author:** ![tebriel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tebriel/32/44889_2.png) [@tebriel](https://discuss.elastic.co/u/tebriel)\
**Post date:** [July 3, 2018, 7:22pm UTC](https://discuss.elastic.co/t/elasticsearch-monitoring-failing/138435/2 "2018-07-03T19:22:38Z")

</div>

`elasticsearch.thread_pool.bulk.rejected` was high, but I was unable to see this as it was renamed to write thread pool ([https://github.com/elastic/elasticsearch/pull/29593](https://github.com/elastic/elasticsearch/pull/29593)). I reverted to the old behavior as my monitoring couldn't see that value ([https://www.elastic.co/guide/en/elasticsearch/reference/6.3/breaking\_63\_settings\_changes.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/breaking_63_settings_changes.html)) and now that my bulk rejections have decreased, data is coming in again.

---

<div class="post-metadata">

**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\
**Post date:** [July 4, 2018, 7:42am UTC](https://discuss.elastic.co/t/elasticsearch-monitoring-failing/138435/3 "2018-07-04T07:42:25Z")

</div>

as this with starting es..  
waiting a minitues .this will ok.

---

<div class="post-metadata">

**Author:** ![tebriel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tebriel/32/44889_2.png) [@tebriel](https://discuss.elastic.co/u/tebriel)\
**Post date:** [July 4, 2018, 3:24pm UTC](https://discuss.elastic.co/t/elasticsearch-monitoring-failing/138435/4 "2018-07-04T15:24:11Z")

</div>

Was the same after 20 minutes and all indexes had recovered and there were no indexes being relocated.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 1, 2018, 3:24pm UTC](https://discuss.elastic.co/t/elasticsearch-monitoring-failing/138435/5 "2018-08-01T15:24:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.