Is an ingest node compulsory for x-pack, in an ES cluster?

[subin]

Hi,

I've an ES cluster of 3 (data and master-eligible nodes) and 1 (coordinating-only) node. I've installed x-pack on all. When I tried to start two nodes, it started throwing up the below:

Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
        at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$null$0(ExportBulk.java:167) ~[?:?]
        ... 50 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
        ... 49 more
Caused by: java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node.
        at org.elasticsearch.action.ingest.IngestActionForwarder.randomIngestNode(IngestActionForwarder.java:58) ~[elasticsearch-6.0.0.jar:6.0.0]
        at org.elasticsearch.action.ingest.IngestActionForwarder.forwardIngestRequest(IngestActionForwarder.java:51) ~[elasticsearch-6.0.0.jar:6.0.0]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:137) ~[elasticsearch-6.0.0.jar:6.0.0]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:86) ~[elasticsearch-6.0.0.jar:6.0.0]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.0.0.jar:6.0.0]
        ... 43 more
[2017-11-27T07:46:14,150][WARN ][o.e.x.m.MonitoringService] [elk02-pco] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
        at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.<init>(ExportBulk.java:106) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1.onFailure(ExportBulk.java:104) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:217) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:211) ~[?:?]
        at org.elasticsearch.xpack.common.IteratingActionListener.onResponse(IteratingActionListener.java:108) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$null$0(ExportBulk.java:175) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:67) ~[elasticsearch-6.0.0.jar:6.0.0]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$1(LocalBulk.java:122) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:67) ~[elasticsearch-6.0.0.jar:6.0.0]

The error ceased only when I added xpack.monitoring.enabled: false in elasticsearch.yml.

Question:

Is a dedicated ingest node necessary for a fully functioning x-pack? If it is not, how can I bring back x-pack monitoring without adding an extra node to the cluster?

Thanks.

PS: In the current 4-node cluster, I've node.ingest: false. The whole case of ingest came up only after I thought to try x-pack. Otherwise, an ingest function isn't required at all for my set up.

[Christian_Dahlqvist]

You do not need a dedicated ingest node, but at least one of your nodes (preferably more) must have the ingest role enabled.

[subin]

Is the coordinating-only node the better candidate in the cluster, to enable ingest on? This one node is where I've Kibana also.

@Christian_Dahlqvist, thanks.

[Christian_Dahlqvist]

As the volume of monitoring data is likely to be low, I would probably enable it on the data nodes to get redundancy and reduce the amount of data that need to be transferred between the nodes. If you see this becoming a problem at some point you can change the role distribution then.