UpdateRequest upsert with retryOnConflict using BulkProcessor failed

Elastic Stack Elasticsearch
1

Althought ES documentation and staff suggests using retry_on_conflict to mitigate version conflict, this feature is broken.

I am using High Level Client 6.6.1 and here is the way I am building the request:

IndexRequest indexRequest = new IndexRequest(MY_INDEX, MY_MAPPING, myId)
            .source(gson.toJson(entity), XContentType.JSON);
        
UpdateRequest updateRequest = new UpdateRequest(MY_INDEX, MY_MAPPING, myId)
            .retryOnConflict(3)
            .doc(indexRequest)
            .upsert(indexRequest);

Without .retryOnConflict(3) it will "work", but some queries will complain about version conflict. I create multiple of those requests and then process it with

client.bulk(request, RequestOptions.DEFAULT);

However when I add .retryOnConflict(3) as shown above it completely breaks with the following stacktrace:

ElasticsearchStatusException[Elasticsearch exception [type=illegal_argument_exception,     reason=Action/metadata line [1] contains an unknown parameter [retry_on_conflict]]]
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2050)
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2026)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1775)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1732)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1694)
        at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:470)

Someone else already posted kind of same question before but unfortunately no response has been provided.

Is there a way to overcome this problem? Should I manually retry the query?

2

Hmm, this looks like a bug, possibly in the HLRC itself. Would you mind opening a ticket in the ES repo?

3

Alright, I will do it. Thanks!

4

Further investigating I am wondering if the problem is actually due to the backend and high level client version mismatch. I have done those requests using High Level Rest Client 6.6.1 but using ES backend 5.5.3. Would you think there could exist such incompatibility?

Thanks!

5

I got similar issue before even though we're using both 6.x server & client, reported on the forum but didn't get any feedback. As I'm using org.elasticsearch.action.bulk.BulkProcessor, for which alternatively you can set the number of retries through BulkProcessor.Builder.setBackoffPolicy() (example at https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-high-document-bulk.html#java-rest-high-document-bulk-processor). However, for you to be aware of, there is another bug on the BulkProcessor reported at BulkProcessor hangs instead of timeout, which doesn't seem resolved yet :frowning:

1 Like
6

Thanks for the info! Unfortunately I am not using BulkProcessor just the High Level client's client.bulk method. Considering what you have said there is not another way around to set a retry policy. I will probably come up with some home-made thing using Failsafe.

7

Possibly... but I would be surprised. The HLRC just emits JSON and uses the standard REST endpoints, and "retry_on_conflict" has been in the REST endpoint for ages. That's actually the main reason we are moving to the HLRC, so that strict versioning becomes a bit more decoupled in java clients (like other languages, python/ruby/php/etc). As long as the JSON is valid for the endpoint the backend and client don't necessarily have to be on the same version.

So I think it's something about the HLRC, either emitting a bad parameter, or unable to parse the response, which is causing the issue.

But.... that's without looking into the issue at all, just a guess :slight_smile:

8

I was finally able to narrow it down. By updating backend to 6.7 the problem has gone. So in my case having a backend 5.5.3 and client 6.7.2 breaks, but when I have aligned both backend and client on 6.7.2 it worked. Some imcompatible behavior is there, but I was not able yet to precisely find it.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.