Infinite bulk-rejection retries via es.batch.write.retry.count = -1

Elastic Stack Elasticsearch
1

The ES-Hadoop configuration documentation states that setting es.batch.write.retry.count to a negative number will direct the ES-Hadoop connector to retry infinitely upon bulk-rejection failures. Could someone advise as to where/how infinite retries are implemented?

Was investigating a Spark app which was failing with "Could not write all entries for bulk operation" EsHadoopException, which would seem to be impossible if the framework were retrying bulk-rejection failures infinitely.

Tracing the es.batch.write.retry.count configuration option appears to show that it is only ever referenced at this location in HttpRetryHandler. In turn, that implementation would appear to result in bulk-rejection failures never retrying rather than retrying indefinitely.

Is there some other explanation I haven't considered? Or is there an actual bug / inconsistency with the documentation for that configuration setting?

2

I'd agree with you here on the inconsistency. It looks like this is a bug that crept in during the rework of the bulk failure handling code when we added failure handlers to the API. I've opened https://github.com/elastic/elasticsearch-hadoop/issues/1178 to track this.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.