Regarding Ingestion Exception

Hi,
I'm seeing an exception during ingestion. Ingestion rates have fallen to very low rates for no apparent reason. Would appreciate some input here:

[2020-10-15T15:46:51,605][WARN ][o.e.x.m.e.l.LocalExporter] [node-A] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[node-B][<ip_address>:9300][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution of primary operation [coordinating_and_primary_bytes=0, replica_bytes=3671438934, all_bytes=3671438934, primary_operation_bytes=2185, max_coordinating_and_primary_bytes=3113851289]];
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.9.1.jar:7.9.1]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) [x-pack-monitoring-7.9.1.jar:7.9.1]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) [x-pack-monitoring-7.9.1.jar:7.9.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.1.jar:7.9.1]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.9.1.jar:7.9.1]

Is this complaining about message payload size? Or something else?

Thx
D

Note that I've been seeing this error for log indexing aswell as monitoring data. So everything appears to be impacted.

It seems that the cluster is at capacity and cannot sustain more writes, that's what the ESRejectedExecutionException hints to. You can take a look at the nodes stats and check if you got increasing rejections in your thread pools.