Issues with task status of enrichment policy execution

Elastic Stack Elasticsearch
1

I'm trying to create an enrichment policy that looks something like

{
    "match": {
        "indices": "index_name",
        "match_field": "unique_id",
        "enrich_fields": ["extra_properties"]
    }
}

Where extra_properties is a flattened field.

When I execute such policy with wait_for_completion=False and start polling the task I get a rather strange response that seems like this:

{'completed': True,
 'task': {'node': '...',
  'id': 1309861,
  'type': 'enrich',
  'action': 'policy_execution',
  'status': {'phase': 'RUNNING', 'step': 'ReindexRequest'},
   .....
}

The task is marked as completed but the status is still running.
When I try to look at the internal .enrich index it seems stuck and not grow.
Sometimes, when I try to manually execute the policy from Kibana it does work and the .enrich policy managed to be created.

When looking in the logs I see this one that looks related:

[tiebreaker-0000000020] failed to execute policy [policy_name] org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:404) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1359) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.4.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1583) ~[?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [instance-0000000023][IP:PORT][indices:data/read/search[phase/query]] disconnected

Looking at this node the CPU usage is at around 30% and memory pressure is at around 20%.

I'm unsure how to handle and treat this case and would love any assistance.