My watcher runs twice

[Khaled_Saidi]

Hi everyone,
i'm trying to figure out what happens with my watcher. since i upgrade my elasticsearch cluster from 7.x to 8.8.2, its runs twice. First time with a success result, the second time error state with this exception:

"version_conflict_engine_exception : required seqNo [35718827], primary term [39]. current document has seqNo [35782663] and primary term [39]".

Is anyone have an idea about this behaviour ?
Thx for your help

[Keith_Massey]

Does restarting the node(s) fix the problem? If so, it might be this problem that we have seen, but only very rarely -- Watcher executes on a node after the .watches shard has been moved to a different node · Issue #105933 · elastic/elasticsearch · GitHub. We have not found a way to reproduce it yet, so tracking down the problem has been difficult. If you are able to reproduce it, I would really appreciate instructions.
If restarting the node(s) does not fix the problem, could you share your elasticsearch logs?

[Khaled_Saidi]

Hi Keith,
Since 3 days, it works fine again... Without any action...

[Keith_Massey]

You could check the logs of all of your nodes to see which one(s) logged that they were running watches, or enabling or disabling watcher, or changing watch shard allocations. But since I have no idea what the problem might have been, unfortunately I can't really tell you what specifically to look for.