Hello,
I am running into an issue when executing an enrich policy via the dotnet NEST API. The top-level issue is that there is no, ".enrich-my-index-name" after issuing the execute on the enrich policy.
There is automation that runs on a once-per-day cadence, this automation does the following actions:
- Creates a new index and batch uploads documents to this newly created index.
- Updates several index policies (max window size, number of replicas, and other fairly innocuous settings)
- Aliases the newly created index.
- Issues a delete request to any other index that was previously being referenced by the alias.
- Executes an associated enrich policy.
Number 5 is executed with the option: wait_for_competion (reason being the code is hosted in an Azure Function which has time limits on how long it can be alive).
The Azure Function reports back success at the end of the series of actions. Although after checking the indices listed in the cluster, there is no enrich index associated with the policy. Furthermore, there is an index, unexpected with same schema as the intended enrich index.
When digging through the logs, I was able to come across this stack trace. Hopefully someone with a bit more knowledge can make sense of what may be going wrong here. If not, my next step is to turn logging to debug - assuming this will grant more fidelity.
[2023-01-03T17:57:58,196][WARN ][o.e.x.e.EnrichPolicyRunner] [junodev01elasticsearch] Policy [node_enrich_policy]: encountered [10000] bulk failures. Turn on DEBUG logging for details.
[2023-01-03T17:57:58,196][ERROR][o.e.x.e.a.InternalExecutePolicyAction] [junodev01elasticsearch] failed to execute policy [node_enrich_policy]
org.elasticsearch.ElasticsearchException: Encountered bulk failures during reindex process
at org.elasticsearch.xpack.enrich.EnrichPolicyRunner$1.onResponse(EnrichPolicyRunner.java:452) [x-pack-enrich-7.17.6.jar:7.17.6]
at org.elasticsearch.xpack.enrich.EnrichPolicyRunner$1.onResponse(EnrichPolicyRunner.java:429) [x-pack-enrich-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.lambda$finishHim$3(AbstractAsyncBulkByScrollAction.java:568) [reindex-client-7.17.6.jar:7.17.6]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.index.reindex.ClientScrollableHitSource.cleanup(ClientScrollableHitSource.java:129) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.index.reindex.ScrollableHitSource.lambda$close$2(ScrollableHitSource.java:115) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.index.reindex.ClientScrollableHitSource$2.onResponse(ClientScrollableHitSource.java:116) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.index.reindex.ClientScrollableHitSource$2.onResponse(ClientScrollableHitSource.java:112) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.ClearScrollController.onFreedContext(ClearScrollController.java:133) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$2(ClearScrollController.java:117) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1471) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1549) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1529) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$3(SearchTransportService.java:454) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:341) [x-pack-security-7.17.6.jar:7.17.6]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:417) [x-pack-security-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:1037) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService.access$100(TransportService.java:66) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:143) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:975) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206) [x-pack-security-7.17.6.jar:7.17.6]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53) [x-pack-security-7.17.6.jar:7.17.6]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167) [x-pack-security-7.17.6.jar:7.17.6]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.SearchTransportService.sendFreeContext(SearchTransportService.java:130) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$4(ClearScrollController.java:114) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.SearchScrollAsyncAction.collectNodesAndRun(SearchScrollAsyncAction.java:116) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.ClearScrollController.cleanScrollIds(ClearScrollController.java:106) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.ClearScrollController.lambda$new$1(ClearScrollController.java:72) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.ClearScrollController.run(ClearScrollController.java:81) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.TransportClearScrollAction.doExecute(TransportClearScrollAction.java:39) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.search.TransportClearScrollAction.doExecute(TransportClearScrollAction.java:19) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:186) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:145) [x-pack-security-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:161) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.client.support.AbstractClient.clearScroll(AbstractClient.java:637) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.index.reindex.ClientScrollableHitSource.clearScroll(ClientScrollableHitSource.java:112) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.index.reindex.ScrollableHitSource.close(ScrollableHitSource.java:115) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.finishHim(AbstractAsyncBulkByScrollAction.java:560) [reindex-client-7.17.6.jar:7.17.6]
at org.elasticsearch.reindex.Reindexer$AsyncIndexBySearchAction.finishHim(Reindexer.java:244) [reindex-client-7.17.6.jar:7.17.6]
at org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.refreshAndFinish(AbstractAsyncBulkByScrollAction.java:522) [reindex-client-7.17.6.jar:7.17.6]
at org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction.onBulkResponse(AbstractAsyncBulkByScrollAction.java:472) [reindex-client-7.17.6.jar:7.17.6]
at org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$2.onResponse(AbstractAsyncBulkByScrollAction.java:421) [reindex-client-7.17.6.jar:7.17.6]
at org.elasticsearch.reindex.AbstractAsyncBulkByScrollAction$2.onResponse(AbstractAsyncBulkByScrollAction.java:417) [reindex-client-7.17.6.jar:7.17.6]
at org.elasticsearch.action.bulk.Retry$RetryHandler.finishHim(Retry.java:168) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.bulk.Retry$RetryHandler.onResponse(Retry.java:112) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.bulk.Retry$RetryHandler.onResponse(Retry.java:71) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:625) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:620) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:97) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1041) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:845) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:1021) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:331) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.postAdded(ClusterStateObserver.java:226) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.cluster.service.ClusterApplierService$1.run(ClusterApplierService.java:272) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.6.jar:7.17.6]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.6.jar:7.17.6]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]