Maximum timeout reached while retrying request. Call: Status code unknown from

Elastic Stack Elasticsearch
1

Hi,

I have installed Elasticsearch 7.17.7 version on Windows Server 2022
I am facing the following error and exception

Index goes to red zone after this exception.
In Elasticsearch logs it's says that that indices files are being used by some other processor, while there is not any other process or a thing which is using it's files.
Some records gets indexed other gets failed.

Elasticsearch.Net.ElasticsearchClientException: Maximum timeout reached while retrying request. Call: Status code unknown from: PUT /indices/_doc/1?pretty=true&error_trace=true&routing=route&refresh=true ---> System.Net.WebException: The operation has timed out
   at System.Net.HttpWebRequest.GetResponse()
   at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)
   --- End of inner exception stack trace ---Debug Information : Invalid NEST response built from a unsuccessful () low level call on PUT: /indices/_doc/1?pretty=true&error_trace=true&routing=route&refresh=true
# Audit trail of this API call:
 - [1] BadResponse: Node: http://endpoint:9200/ Took: 00:01:00.0155036
 - [2] MaxTimeoutReached:
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: Maximum timeout reached while retrying request. Call: Status code unknown from: PUT /indices/_doc/1?pretty=true&error_trace=true&routing=route&refresh=true ---> System.Net.WebException: The operation has timed out
   at System.Net.HttpWebRequest.GetResponse()
   at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)
   --- End of inner exception stack trace ---
#Request
# Response:

# TCP states:
  TimeWait: 1778
  Established: 733
  CloseWait: 24
  FinWait2: 4

# ThreadPool statistics:
  Worker: 
    Busy: 2
    Free: 8189
    Min: 32
    Max: 8191
  IOCP: 
    Busy: 0
    Free: 1000
    Min: 32
    Max: 1000

Elasticsearch Server Logs exception are following :

{"type": "server", "timestamp": "2023-10-18T01:30:07,588-04:00", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "elasticsearch", "node.name": "CUREMD-INT02", "message": "triggering scheduled [ML] maintenance tasks", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA"  }
{"type": "server", "timestamp": "2023-10-18T01:30:07,605-04:00", "level": "INFO", "component": "o.e.x.m.a.TransportDeleteExpiredDataAction", "cluster.name": "elasticsearch", "node.name": "CUREMD-INT02", "message": "Deleting expired data", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA"  }
{"type": "server", "timestamp": "2023-10-18T01:30:07,607-04:00", "level": "INFO", "component": "o.e.x.m.j.r.UnusedStatsRemover", "cluster.name": "elasticsearch", "node.name": "CUREMD-INT02", "message": "Successfully deleted [0] unused stats documents", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA"  }
{"type": "server", "timestamp": "2023-10-18T01:30:07,608-04:00", "level": "INFO", "component": "o.e.x.m.a.TransportDeleteExpiredDataAction", "cluster.name": "elasticsearch", "node.name": "CUREMD-INT02", "message": "Completed deletion of expired ML data", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA"  }
{"type": "server", "timestamp": "2023-10-18T01:30:07,608-04:00", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "elasticsearch", "node.name": "CUREMD-INT02", "message": "Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA"  }
{"type": "server", "timestamp": "2023-10-18T21:08:09,483-04:00", "level": "WARN", "component": "o.e.i.e.Engine", "cluster.name": "elasticsearch", "node.name": "CUREMD-INT02", "message": " [interfacehub][86] failed engine [lucene commit failed]", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA" , 
"stacktrace": ["java.nio.file.FileSystemException: D:\\InterfaceHub\\ElasticSearch\\DataFiles\\nodes\\0\\indices\\hOQYM3X-Qm-bQUtjYTi0rA\\86\\index\\_2niv.cfs: The process cannot access the file because it is being used by another process.",

The logs I received today.

{"type": "server", "timestamp": "2024-01-04T03:28:10,864-05:00", "level": "WARN", "component": "o.e.x.t.t.TransformIndexer", "cluster.name": "elasticsearch", "node.name": "indices", "message": "[endpoint.metadata_united-default-1.3.0] failed to detect changes for transform. Skipping update till next check.", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA" , 
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:713) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:400) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:745) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:64) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:331) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:651) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService$4.handleException(TransportService.java:853) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1481) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:1016) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:288) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:990) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167) ~[?:?]",
"at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:937) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:925) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:250) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:104) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:316) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:352) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:244) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:199) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:186) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) ~[?:?]",
"at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:128) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:103) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:100) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:550) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:444) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:961) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:925) ~[?:?]",
"at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:361) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1009) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:352) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:437) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:374) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:259) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:147) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:173) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:279) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:153) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:261) ~[?:?]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182) ~[?:?]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) ~[?:?]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:161) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:55) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:251) ~[?:?]",
"at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:211) ~[?:?]",
"at org.elasticsearch.xpack.transform.checkpoint.TimeBasedCheckpointProvider.sourceHasChanged(TimeBasedCheckpointProvider.java:80) ~[?:?]",
"at org.elasticsearch.xpack.transform.transforms.TransformIndexer.sourceHasChanged(TransformIndexer.java:1017) ~[?:?]",
"at org.elasticsearch.xpack.transform.transforms.TransformIndexer.onStart(TransformIndexer.java:367) ~[?:?]",
"at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$maybeTriggerAsyncJob$5(AsyncTwoPhaseIndexer.java:219) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.7.jar:7.17.7]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_261]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_261]",
"at java.lang.Thread.run(Unknown Source) [?:1.8.0_261]",
"Caused by: org.elasticsearch.node.NodeClosedException: node closed {CUREMD-INT02}{rlLmWtXIQYSY5yxGWm3UzA}{egqUGTsjSw-wxIYeRPRJrg}{10.173.14.131}{10.173.14.131:9300}{cdfhilmrstw}{ml.machine_memory=136912175104, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=8109031424}",
"at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:969) ~[elasticsearch-7.17.7.jar:7.17.7]",
"... 102 more"] }
{"type": "server", "timestamp": "2024-01-04T03:28:14,864-05:00", "level": "WARN", "component": "o.e.x.t.t.TransformIndexer", "cluster.name": "elasticsearch", "node.name": "CUREMD-INT02", "message": "[endpoint.metadata_united-default-1.3.0] failed to detect changes for transform. Skipping update till next check.", "cluster.uuid": "bOVT1A1MSsO0-gcEU11vXg", "node.id": "rlLmWtXIQYSY5yxGWm3UzA" , 
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:713) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:400) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:745) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:64) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:331) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:651) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService$4.handleException(TransportService.java:853) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1481) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:1016) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:288) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:990) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167) ~[?:?]",
"at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:937) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:925) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:250) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:104) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:316) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:352) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:244) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:199) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:186) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) ~[?:?]",
"at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:571) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:128) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:103) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:100) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:565) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:550) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:444) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:961) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:925) ~[?:?]",
"at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:361) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1009) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:352) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:437) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:374) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:259) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:147) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:173) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:279) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:153) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:261) ~[?:?]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182) ~[?:?]",
"at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) ~[?:?]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:161) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:55) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.7.jar:7.17.7]",
"at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:251) ~[?:?]",
"at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:211) ~[?:?]",
"at org.elasticsearch.xpack.transform.checkpoint.TimeBasedCheckpointProvider.sourceHasChanged(TimeBasedCheckpointProvider.java:80) ~[?:?]",
"at org.elasticsearch.xpack.transform.transforms.TransformIndexer.sourceHasChanged(TransformIndexer.java:1017) ~[?:?]",
"at org.elasticsearch.xpack.transform.transforms.TransformIndexer.onStart(TransformIndexer.java:367) ~[?:?]",
"at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$maybeTriggerAsyncJob$5(AsyncTwoPhaseIndexer.java:219) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.7.jar:7.17.7]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_261]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_261]",
"at java.lang.Thread.run(Unknown Source) [?:1.8.0_261]",
"Caused by: org.elasticsearch.node.NodeClosedException: node closed {CUREMD-INT02}{rlLmWtXIQYSY5yxGWm3UzA}{egqUGTsjSw-wxIYeRPRJrg}{10.173.14.131}{10.173.14.131:9300}{cdfhilmrstw}{ml.machine_memory=136912175104, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=8109031424}",
"at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:969) ~[elasticsearch-7.17.7.jar:7.17.7]",
"... 102 more"] }```
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.