# Intermittent 503 error with Scroll API **URL:** \ **Category:** Elasticsearch\ **Created:** [March 13, 2019, 6:33pm UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199 "2019-03-13T18:33:00Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![BinarMorker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/binarmorker/32/41934_2.png) [@BinarMorker](https://discuss.elastic.co/u/BinarMorker)\ **Post date:** [March 13, 2019, 6:33pm UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/1 "2019-03-13T18:33:00Z")
Hi, I have built a console app in .Net Core that interfaces with ElasticSearch through NEST. I'm trying to send this command: ``` POST /products_en-ca_b2df44e0-2a4e-433e-9ea0-ec8a9e8f9e19/product/_search?pretty=true&error_trace=true&typed_keys=true&scroll=2m { "from": 0, "size": 100, "query": { "match_all": {} } } ``` In my program, it is written as such: ``` await ElasticClient.SearchAsync(search => search .From(0) .Take(100) .MatchAll() .Scroll(ScrollTimeout) ); ``` The issue I'm having is that most of the time, I get a normal response with hits and all. But sometimes, and seemingly at random, I get this response: ``` { "error" : { "root_cause" : [], "type" : "search_phase_execution_exception", "reason" : "all shards failed", "phase" : "query", "grouped" : true, "failed_shards" : [], "stack_trace" : "Failed to execute phase [query], all shards failed\r\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293)\r\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133)\r\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254)\r\n\tat org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101)\r\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48)\r\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222)\r\n\tat org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176)\r\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48)\r\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222)\r\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\r\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)\r\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:464)\r\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1130)\r\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1247)\r\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1221)\r\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66)\r\n\tat org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112)\r\n\tat org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:347)\r\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:341)\r\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)\r\n\tat org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\r\n\tat java.lang.Thread.run(Unknown Source)\r\n" }, "status" : 503 } ``` Here is the stack trace in a better format: ``` Failed to execute phase [query], all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:464) at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1130) at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1247) at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1221) at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:347) at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:341) at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ``` And that's it. I have no clue what is happening. There is no more information about the error. Thanks. ---
**Author:** ![BinarMorker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/binarmorker/32/41934_2.png) [@BinarMorker](https://discuss.elastic.co/u/BinarMorker)\ **Post date:** [March 18, 2019, 11:44am UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/2 "2019-03-18T11:44:15Z")
Bump ---
**Author:** ![xavierfacq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavierfacq/32/8744_2.png) [@xavierfacq](https://discuss.elastic.co/u/xavierfacq)\ **Post date:** [March 18, 2019, 1:42pm UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/3 "2019-03-18T13:42:32Z")
Hi, How many time elapse between 2 call to the scroll API ? bye, Xavier ---
**Author:** ![BinarMorker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/binarmorker/32/41934_2.png) [@BinarMorker](https://discuss.elastic.co/u/BinarMorker)\ **Post date:** [March 18, 2019, 2:10pm UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/4 "2019-03-18T14:10:07Z")
They are very quick. I'm chaining them as fast as possible as I want to get all the content of my index, which can contain an indeterminate number of documents. The two calls are about 80ms to 300ms between each. ---
**Author:** ![xavierfacq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavierfacq/32/8744_2.png) [@xavierfacq](https://discuss.elastic.co/u/xavierfacq)\ **Post date:** [March 19, 2019, 7:27am UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/5 "2019-03-19T07:27:34Z")
And do you well pass the scroll\_id to next queries when srolling ? [https://www.elastic.co/guide/en/elasticsearch/reference/6.6/search-request-scroll.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/search-request-scroll.html) ---
**Author:** ![BinarMorker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/binarmorker/32/41934_2.png) [@BinarMorker](https://discuss.elastic.co/u/BinarMorker)\ **Post date:** [March 19, 2019, 12:35pm UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/6 "2019-03-19T12:35:26Z")
Yes. The first call looks like that, but afterwards I'm passing the scrollId and retrieving it after each call to use on the next. ---
**Author:** ![BinarMorker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/binarmorker/32/41934_2.png) [@BinarMorker](https://discuss.elastic.co/u/BinarMorker)\ **Post date:** [March 22, 2019, 5:18pm UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/7 "2019-03-22T17:18:52Z")
Bump ---
**Author:** ![Natalie964](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@Natalie964](https://discuss.elastic.co/u/Natalie964)\ **Post date:** [March 23, 2019, 10:27am UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/8 "2019-03-23T10:27:38Z")
Http.Sys does have a request queue and it will send 503s if the queue gets full. The RequestQueueLimit option was added in a later version that would let you adjust the length. If the logs indicate your queue is full then consider updating and setting this. [ttrockstars](https://ttrockstars.org/) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 20, 2019, 10:27am UTC](https://discuss.elastic.co/t/intermittent-503-error-with-scroll-api/172199/9 "2019-04-20T10:27:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.