Regarding Ingestion Exception

Elastic Stack Elasticsearch
1

Hi,
I'm seeing an exception during ingestion. Ingestion rates have fallen to very low rates for no apparent reason. Would appreciate some input here:

[2020-10-15T15:46:51,605][WARN ][o.e.x.m.e.l.LocalExporter] [node-A] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[node-B][<ip_address>:9300][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution of primary operation [coordinating_and_primary_bytes=0, replica_bytes=3671438934, all_bytes=3671438934, primary_operation_bytes=2185, max_coordinating_and_primary_bytes=3113851289]];
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.9.1.jar:7.9.1]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) [x-pack-monitoring-7.9.1.jar:7.9.1]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) [x-pack-monitoring-7.9.1.jar:7.9.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.1.jar:7.9.1]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.9.1.jar:7.9.1]

Is this complaining about message payload size? Or something else?

Thx
D

2

Note that I've been seeing this error for log indexing aswell as monitoring data. So everything appears to be impacted.

3

It seems that the cluster is at capacity and cannot sustain more writes, that's what the ESRejectedExecutionException hints to. You can take a look at the nodes stats and check if you got increasing rejections in your thread pools.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.