Monitoring execution failed

Hi ALL,
my cluster is as follow :

[root@es-data-01 ~]# curl -u elastic:****** -XGET http://172.24.69.16:9200/_cat/nodes
172.24.69.20 33 98 2 0.03 0.06 0.07 m * es-master-02
172.24.69.13 19 98 0 0.02 0.02 0.05 - - es-client-01
172.24.69.17 42 98 18 10.45 6.33 3.85 d - es-data-02
172.24.69.21 9 98 0 0.00 0.01 0.05 - - es-client-03
172.24.69.16 17 98 40 17.18 18.91 12.90 d - es-data-01
172.24.69.22 4 88 1 0.00 0.03 0.05 m - es-master-03
172.24.69.14 10 98 1 0.00 0.01 0.05 - - es-client-02
172.24.69.19 13 98 1 0.02 0.05 0.05 m - es-master-01
172.24.69.18 78 99 68 3.58 2.06 1.28 d - es-data-03

I have the following error in my nodes :

[2017-12-10T10:48:05,731][WARN ][o.e.x.m.MonitoringService] [es-master-02] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.(ExportBulk.java:106) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1.onFailure(ExportBulk.java:104) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:217) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:211) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$null$0(ExportBulk.java:175) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:67) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$1(LocalBulk.java:118) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:67) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:67) ~[elasticsearch-5.6.3.jar:5.6.3]
5.6.3.jar:5.6.3]

    at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122) ~[?:?]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:185) ~[?:?]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:145) ~[?:?]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.xpack.security.InternalClient.doExecute(InternalClient.java:101) ~[?:?]
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) ~[elasticsearch-5.6.3.jar:5.6.3]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:112) ~[x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:71) ~[x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$1(ExportBulk.java:165) ~[x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.common.IteratingActionListener.run(IteratingActionListener.java:93) [x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:181) [x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flushAndClose(ExportBulk.java:95) [x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:85) [x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:202) [x-pack-5.6.3.jar:?]
    at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:205) [x-pack-5.6.3.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_111]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.3.jar:5.6.3]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$null$0(ExportBulk.java:167) ~[?:?]
... 51 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
... 50 more
Caused by: java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node.
at org.elasticsearch.action.ingest.IngestActionForwarder.randomIngestNode(IngestActionForwarder.java:58) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.ingest.IngestActionForwarder.forwardIngestRequest(IngestActionForwarder.java:51) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:138) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:85) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.3.jar:5.6.3]
... 43 more

would you please help me to fix it
best

There is a tiny part in this exception revealing the cause

which means none of the nodes in your cluster do have the ingest role, and thus indexing does not work. Is there any specific reason you disabled this?

--Alex

hi
as mentioned before I have 3 ingest node that work as coordinator and Logstashs send all logs to this 3 ingest node and this nodes distribute logs to my cluster
would you please help me
best

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.