Hi Team,
We are using transforms in our cluster and we have applied retention in the transforms for 90 days. But now a days we are frequently seeing below warnings in transforms. Could you please help me to understand the issues.
Warnings: "Transform encountered an exception: [org.elasticsearch.xpack.transform.transforms.RetentionPolicyToDeleteByQueryRequestConverter$RetentionPolicyException: found [579] version conflicts when deleting documents as part of the retention policy.]; Will automatically retry [1/10]"
Thanks in Advance.
Transforms uses the DeleteByQuery API to manage the retention policy, and this error comes from that API: Delete by query API | Elasticsearch Guide [8.15] | Elastic
If a document changes between the time that the snapshot is taken and the delete operation is processed, it results in a version conflict and the delete operation fails.
For Transforms, the DeleteByQuery API is called immediately following a refresh of the destination index, so the delete should be on the most up-to-date info, but in any case this warning is usually benign in that it will retry automatically.
In this case, is the Transforms eventually failing and stopping? Is there another Transform or process writing to the same destination index? What version of Elasticsearch is being used?