# All shards failed exception and Elasticsearch service stopped

**URL:** <https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961>\
**Category:** Elasticsearch\
**Created:** [December 1, 2017, 4:34pm UTC](https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961 "2017-12-01T16:34:37Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![iluvcode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iluvcode/32/24929_2.png) [@iluvcode](https://discuss.elastic.co/u/iluvcode)\
**Post date:** [December 1, 2017, 4:34pm UTC](https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961/1 "2017-12-01T16:34:38Z")

</div>

Hi,  
Elasticsearch 5.3 service installed in our production server stopped by itself and after looking at the log files, i found below information. since the query is requesting 85580 records, all shards failed and elasticsearch shut itself down? if that's the reason, how can i fix this issue ? i looked at "max\_result\_window" and does setting the size to 10k will help ?

```
[2017-11-30T13:34:09,019][WARN][r.suppressed] path: /elasticsearchlive/searchentry/_search, params: {index=elasticsearchlive, type=searchentry}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onInitialPhaseResult(AbstractSearchAsyncAction.java:223) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:58) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:148) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1032) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1134) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1112) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:629) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:598) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.3.0.jar:5.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.transport.RemoteTransportException: [web1][10.100.6.2:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Result window is too large, from + size must be less than or equal to: [10000] but was [85580]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.
	at org.elasticsearch.search.DefaultSearchContext.preProcess(DefaultSearchContext.java:202) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:90) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:480) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:444) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:331) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:328) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:618) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:613) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.3.0.jar:5.3.0]
	... 3 more
[2017-11-30T15:31:03,379][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [web1] fatal error in thread [elasticsearch[web1][search][T#25]], exiting
java.lang.StackOverflowError: null
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1309) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
```

---

<div class="post-metadata">

**Author:** ![luiz.santos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luiz.santos/32/24664_2.png) [@luiz.santos](https://discuss.elastic.co/u/luiz.santos)\
**Post date:** [December 8, 2017, 12:29pm UTC](https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961/2 "2017-12-08T12:29:41Z")

</div>

Hi @iluvcode,

I think we two event here. The first one happened [2017-11-30T13:34:09,019] and is just a WARN. This warn means that you are trying to paginate too deep. Typically when parameters [from + size used exceed 10.000](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html) (by default). If you want to retrieve 85580 documents maybe you will interested in the [scroll api](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll) or [search after](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html).

The second event happened about two hours latter [2017-11-30T15:31:03,379] and caused the crash. It's not clear for me what might have cause this fatal error. Please, if you have more evidences feel free to post here and maybe we can determine the root cause.

Cheers,  
LG

---

<div class="post-metadata">

**Author:** ![iluvcode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iluvcode/32/24929_2.png) [@iluvcode](https://discuss.elastic.co/u/iluvcode)\
**Post date:** [December 8, 2017, 7:35pm UTC](https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961/3 "2017-12-08T19:35:22Z")

</div>

@luiz.santos : apart from the events I posted, i don't have any other evidences. Same issue happened in other production server this week. is there a way find out why Elasticsearch service is crashing ?

Thanks

---

<div class="post-metadata">

**Author:** ![luiz.santos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luiz.santos/32/24664_2.png) [@luiz.santos](https://discuss.elastic.co/u/luiz.santos)\
**Post date:** [December 8, 2017, 7:47pm UTC](https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961/4 "2017-12-08T19:47:47Z")

</div>

Hi @iluvcode,

Some important informations are:

- elasticsearch version
- how are using elasticsearch
- number of indices
- total number of shards
- most frequent operation
- any suspect query?
- is your cluster suffering of out of memory?
- etc

Did you followed the [production guides](https://www.elastic.co/guide/en/elasticsearch/guide/current/deploy.html)?

Cheers,  
LG

---

<div class="post-metadata">

**Author:** ![iluvcode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iluvcode/32/24929_2.png) [@iluvcode](https://discuss.elastic.co/u/iluvcode)\
**Post date:** [December 11, 2017, 4:02pm UTC](https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961/5 "2017-12-11T16:02:55Z")

</div>

- elasticsearch version  
Elasticsearch 5.3.0
- how are using elasticsearch  
We are using elasticsearch for our public facing ecommerce search.
- number of indices (Index information)  
"elasticsearchlive": {  
"primaries": {  
"docs": {  
"count": 1378271,  
"deleted": 637080  
},  
"store": {  
"size\_in\_bytes": 20084703077,  
"throttle\_time\_in\_millis": 0  
},  
"indexing": {  
"index\_total": 9691855,  
"index\_time\_in\_millis": 20032575,  
"index\_current": 0,  
"index\_failed": 0,  
"delete\_total": 1601911,  
"delete\_time\_in\_millis": 102978,  
"delete\_current": 0,  
"noop\_update\_total": 0,  
"is\_throttled": false,  
"throttle\_time\_in\_millis": 0  
},  
"get": {  
"total": 0,  
"time\_in\_millis": 0,  
"exists\_total": 0,  
"exists\_time\_in\_millis": 0,  
"missing\_total": 0,  
"missing\_time\_in\_millis": 0,  
"current": 0  
},  
"search": {  
"open\_contexts": 0,  
"query\_total": 157221949,  
"query\_time\_in\_millis": 94346276,  
"query\_current": 0,  
"fetch\_total": 51648965,  
"fetch\_time\_in\_millis": 30139609,  
"fetch\_current": 0,  
"scroll\_total": 8284560,  
"scroll\_time\_in\_millis": 7317305,  
"scroll\_current": 0,  
"suggest\_total": 19167516,  
"suggest\_time\_in\_millis": 18501423,  
"suggest\_current": 0  
},  
"merges": {  
"current": 0,  
"current\_docs": 0,  
"current\_size\_in\_bytes": 0,  
"total": 42711,  
"total\_time\_in\_millis": 42241303,  
"total\_docs": 108222518,  
"total\_size\_in\_bytes": 297588814453,  
"total\_stopped\_time\_in\_millis": 0,  
"total\_throttled\_time\_in\_millis": 15174382,  
"total\_auto\_throttle\_in\_bytes": 26214400  
},  
"refresh": {  
"total": 390096,  
"total\_time\_in\_millis": 46504445,  
"listeners": 0  
},  
"flush": {  
"total": 2022,  
"total\_time\_in\_millis": 444978  
},  
"warmer": {  
"current": 0,  
"total": 391854,  
"total\_time\_in\_millis": 174148  
},  
"query\_cache": {  
"memory\_size\_in\_bytes": 238823384,  
"total\_count": 1615358046,  
"hit\_count": 267504494,  
"miss\_count": 1347853552,  
"cache\_size": 74725,  
"cache\_count": 3414241,  
"evictions": 3339516  
},  
"fielddata": {  
"memory\_size\_in\_bytes": 0,  
"evictions": 0  
},  
"completion": {  
"size\_in\_bytes": 102963592  
},  
"segments": {  
"count": 137,  
"memory\_in\_bytes": 118387711,  
"terms\_memory\_in\_bytes": 112327761,  
"stored\_fields\_memory\_in\_bytes": 623184,  
"term\_vectors\_memory\_in\_bytes": 0,  
"norms\_memory\_in\_bytes": 368832,  
"points\_memory\_in\_bytes": 1301218,  
"doc\_values\_memory\_in\_bytes": 3766716,  
"index\_writer\_memory\_in\_bytes": 0,  
"version\_map\_memory\_in\_bytes": 99111,  
"fixed\_bit\_set\_memory\_in\_bytes": 257968,  
"max\_unsafe\_auto\_id\_timestamp": -1,  
"file\_sizes": {}  
},  
"translog": {  
"operations": 103654,  
"size\_in\_bytes": 380121790  
},  
"request\_cache": {  
"memory\_size\_in\_bytes": 0,  
"evictions": 0,  
"hit\_count": 30359,  
"miss\_count": 8242  
},  
"recovery": {  
"current\_as\_source": 0,  
"current\_as\_target": 0,  
"throttle\_time\_in\_millis": 0  
}  
},  
"total": {  
"docs": {  
"count": 1378271,  
"deleted": 637080  
},  
"store": {  
"size\_in\_bytes": 20084703077,  
"throttle\_time\_in\_millis": 0  
},  
"indexing": {  
"index\_total": 9691855,  
"index\_time\_in\_millis": 20032575,  
"index\_current": 0,  
"index\_failed": 0,  
"delete\_total": 1601911,  
"delete\_time\_in\_millis": 102978,  
"delete\_current": 0,  
"noop\_update\_total": 0,  
"is\_throttled": false,  
"throttle\_time\_in\_millis": 0  
},  
"get": {  
"total": 0,  
"time\_in\_millis": 0,  
"exists\_total": 0,  
"exists\_time\_in\_millis": 0,  
"missing\_total": 0,  
"missing\_time\_in\_millis": 0,  
"current": 0  
},  
"search": {  
"open\_contexts": 0,  
"query\_total": 157221949,  
"query\_time\_in\_millis": 94346276,  
"query\_current": 0,  
"fetch\_total": 51648965,  
"fetch\_time\_in\_millis": 30139609,  
"fetch\_current": 0,  
"scroll\_total": 8284560,  
"scroll\_time\_in\_millis": 7317305,  
"scroll\_current": 0,  
"suggest\_total": 19167516,  
"suggest\_time\_in\_millis": 18501423,  
"suggest\_current": 0  
},  
"merges": {  
"current": 0,  
"current\_docs": 0,  
"current\_size\_in\_bytes": 0,  
"total": 42711,  
"total\_time\_in\_millis": 42241303,  
"total\_docs": 108222518,  
"total\_size\_in\_bytes": 297588814453,  
"total\_stopped\_time\_in\_millis": 0,  
"total\_throttled\_time\_in\_millis": 15174382,  
"total\_auto\_throttle\_in\_bytes": 26214400  
},  
"refresh": {  
"total": 390096,  
"total\_time\_in\_millis": 46504445,  
"listeners": 0  
},  
"flush": {  
"total": 2022,  
"total\_time\_in\_millis": 444978  
},  
"warmer": {  
"current": 0,  
"total": 391854,  
"total\_time\_in\_millis": 174148  
},  
"query\_cache": {  
"memory\_size\_in\_bytes": 238823384,  
"total\_count": 1615358046,  
"hit\_count": 267504494,  
"miss\_count": 1347853552,  
"cache\_size": 74725,  
"cache\_count": 3414241,  
"evictions": 3339516  
},  
"fielddata": {  
"memory\_size\_in\_bytes": 0,  
"evictions": 0  
},  
"completion": {  
"size\_in\_bytes": 102963592  
},  
"segments": {  
"count": 137,  
"memory\_in\_bytes": 118387711,  
"terms\_memory\_in\_bytes": 112327761,  
"stored\_fields\_memory\_in\_bytes": 623184,  
"term\_vectors\_memory\_in\_bytes": 0,  
"norms\_memory\_in\_bytes": 368832,  
"points\_memory\_in\_bytes": 1301218,  
"doc\_values\_memory\_in\_bytes": 3766716,  
"index\_writer\_memory\_in\_bytes": 0,  
"version\_map\_memory\_in\_bytes": 99111,  
"fixed\_bit\_set\_memory\_in\_bytes": 257968,  
"max\_unsafe\_auto\_id\_timestamp": -1,  
"file\_sizes": {}  
},  
"translog": {  
"operations": 103654,  
"size\_in\_bytes": 380121790  
},  
"request\_cache": {  
"memory\_size\_in\_bytes": 0,  
"evictions": 0,  
"hit\_count": 30359,  
"miss\_count": 8242  
},  
"recovery": {  
"current\_as\_source": 0,  
"current\_as\_target": 0,  
"throttle\_time\_in\_millis": 0  
}  
}  
}
- most frequent operation  
Search query
- any suspect query?  
None
- is your cluster suffering of out of memory?  
Our production server is using SSD drive with 120GB of memory and we allocated 30GB to Elasticsearch.

Yes, I did follow Production guides. we never had this issue when using Elasticsearch 1.7.1. After upgrading elasticsearch to 5.3, i started seeing this issue.  
Please let me know if you need more information.

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 8, 2018, 4:03pm UTC](https://discuss.elastic.co/t/all-shards-failed-exception-and-elasticsearch-service-stopped/109961/6 "2018-01-08T16:03:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.