VersionConflictEngineException throw after the upgrade From 5.0.0 to 5.6.16

dalei2019 · October 25, 2021, 12:31pm

Sorry, I known it is old version of ES .
Our plan is upgrade old version of es cluster to 7.15.x .
Now after first step, upgrade to 5.6 , concurrent index write use bulkRequest (transport client ) throw many VersionConflictEngineException (1 error in one min).

[T13:07:03,235][TRACE][o.e.a.b.TransportShardBulkAction] [fht-pro-node5] [xxxx][0] failed to execute bulk item (update) BulkShardRequest [[xxxx][0]] containing [org.elasticsearch.action.update.UpdateRequest@7ae5265f] and a refresh
org.elasticsearch.index.engine.VersionConflictEngineException: [xxxx][yyyy]: version conflict, current version [2] is different than the one provided [1]
        at org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:582) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:493) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:557) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:493) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:290) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:161) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
[T13:07:03,237][TRACE][o.e.a.b.TransportShardBulkAction] [fht-pro-node5] [[xxxx][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[xxxx][0]] containing [org.elasticsearch.action.update.UpdateRequest@7ae5265f] and a refresh]
[T13:07:03,237][TRACE][o.e.a.b.TransportShardBulkAction] [fht-pro-node5] [[xxxx][0]] sending op [indices:data/write/bulk[s]] to replica [xxxx][0], node[lR_jsdeYTj2KGuuK-GqRYg], [R], s[STARTED], a[id=1bUlk32RTE2U9sS2WntD2A] for request [BulkShardRequest [[xxxx][0]] containing [org.elasticsearch.action.update.UpdateRequest@7ae5265f] and a refresh]
[T13:07:03,239][TRACE][o.e.a.b.TransportShardBulkAction] [fht-pro-node5] [[xxxx][0]] op [indices:data/write/bulk[s]] completed on primary for reque

Before upgrade , everything just fine.No VersionConflictEngineException Throw.
I checked each service ， No obvious bugs found. Try increase retryOnConflict from 3 to 5 ，no significant effect.

The load did not increase during the upgrade or after upgrade.
I have spent a week troubleshooting the issue， But no progress .I was under a lot of pressure.

My Question:

Does version control or index logic changed between 5.0 and 5.6 ?
How can I find this exact reason of this fault , Any advice or solutions?

Anyone can help me , thanks .

warkolm · October 25, 2021, 10:08pm

I am not sure you will be able to get much advice on changes between 5.0 and 5.6, as it's been 5 years since release, and they are at least 2 years past EOL.

I would suggest creating a new 7.15 cluster, then running a remote reindex from the old cluster and then see how your code works with that.

dalei2019 · October 26, 2021, 3:11am

Thanks for your reply , warkolm.
Direct upgrade to 7.15 , need lot of work to do .
Because many app build on es , and many data , may be need 1-2 months work to migrate .
Now i am stuck ES 5.6 , because of this exception .

Has anyone encountered a similar problem, I would be very grateful if you can provide some help .

system · November 23, 2021, 3:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.