ElasticSearch Monitoring Failing

tebriel · July 3, 2018, 5:48pm

Upgraded to 6.3.0, elasticsearch monitoring now fails, stack trace follows. I'm seeing occasional bursts of data showing up in Kibana 6.3.0, but it's got gaps in data. I tried increasing shards to 3 (from the 1 in the template) and deleting the index, no improvement.
Monitoring is enabled via the persistent settings API:

{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

[2018-07-03T17:43:08,357][WARN ][o.e.x.m.MonitoringService] [m0_nP62] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.<init>(ExportBulk.java:95) ~[x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1.onFailure(ExportBulk.java:93) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:206) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:200) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:96) [x-pack-core-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:164) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:68) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:135) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:111) [x-pack-monitoring-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:570) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:379) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:374) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:897) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:855) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.lambda$handleException$34(TcpTransport.java:1510) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1508) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1500) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1430) [elasticsearch-6.3.0.jar:6.3.0]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) [transport-netty4-6.3.0.jar:6.3.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
<snip to be under max char count>
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
	at java.lang.Thread.run(Thread.java:844) [?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
	at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:156) [x-pack-monitoring-6.3.0.jar:6.3.0]
	... 46 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:124) ~[?:?]
	... 44 more

tebriel · July 3, 2018, 7:22pm

elasticsearch.thread_pool.bulk.rejected was high, but I was unable to see this as it was renamed to write thread pool (https://github.com/elastic/elasticsearch/pull/29593). I reverted to the old behavior as my monitoring couldn't see that value (https://www.elastic.co/guide/en/elasticsearch/reference/6.3/breaking_63_settings_changes.html) and now that my bulk rejections have decreased, data is coming in again.

zqc0512 · July 4, 2018, 7:42am

as this with starting es..
waiting a minitues .this will ok.

tebriel · July 4, 2018, 3:24pm

Was the same after 20 minutes and all indexes had recovered and there were no indexes being relocated.

system · August 1, 2018, 3:24pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
After restarting Elasticsearch, monitoring execution fails Elasticsearch	2	5365	July 7, 2017
Monitoring Not working not sure about the cause Kibana elastic-stack-monitoring	2	1915	March 1, 2019
Elastic stuck and flooded with "monitoring execution failed" Elasticsearch	10	860	October 7, 2020
Couldn't Activate Monitoring in 6.3 Kibana	2	6513	July 13, 2018
Monitoring execution failed Elasticsearch	3	9700	January 9, 2018

ElasticSearch Monitoring Failing

Related topics