hi can anyone help me
I have 3 node cluster and I get this error when trying to get a large data set of more than 250,000 documents with search_after requests, the batch size is 10,000 documents
i'm using nest .net client and my connection settings are as follows
var connectionSettings = new ConnectionSettings(connectionPool, sourceSerializer: JsonNetSerializer.Default)
.DefaultIndex(alias)
.EnableHttpCompression()
.ThrowExceptions()
.ConnectionLimit(1000)
.MaximumRetries(20)
error log is
[2019-12-16T08:15:10,339][DEBUG][o.e.a.s.TransportSearchAction] [data-node-2] [1492551] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [data-node-1][10.0.0.4:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:149) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:502) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$runAsync$2(SearchService.java:344) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.4.2.jar:7.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2019-12-16T08:15:10,340][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [data-node-2] Sending remove ban for tasks with the parent [dpufuE4mSqWnPrEnCBF6qw:3531195] to the node [XdjH8gUmSfOT40UKSd4OKg]
[2019-12-16T08:15:10,340][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [data-node-2] Sending remove ban for tasks with the parent [dpufuE4mSqWnPrEnCBF6qw:3531195] to the node [H4lPnGn7T-OKmPlwlsZUkg]
[2019-12-16T08:15:10,341][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [data-node-2] Sending remove ban for tasks with the parent [dpufuE4mSqWnPrEnCBF6qw:3531195] to the node [dpufuE4mSqWnPrEnCBF6qw]
[2019-12-16T08:15:10,341][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [data-node-2] Removing ban for the parent [dpufuE4mSqWnPrEnCBF6qw:3531195] on the node [dpufuE4mSqWnPrEnCBF6qw]