# All Shards Failing **URL:** \ **Category:** Elasticsearch\ **Created:** [June 26, 2019, 4:23pm UTC](https://discuss.elastic.co/t/all-shards-failing/187626 "2019-06-26T16:23:19Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Analyst](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@Analyst](https://discuss.elastic.co/u/Analyst)\ **Post date:** [June 26, 2019, 4:23pm UTC](https://discuss.elastic.co/t/all-shards-failing/187626/1 "2019-06-26T16:23:19Z")
Hi Team, I was trying to delete the old indices and ran into an issue. Now i am getting the below message in elastic search logs? Could you please help me out here? ---
**Author:** ![Analyst](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@Analyst](https://discuss.elastic.co/u/Analyst)\ **Post date:** [June 26, 2019, 4:23pm UTC](https://discuss.elastic.co/t/all-shards-failing/187626/2 "2019-06-26T16:23:47Z")
[2019-06-26T14:07:21,613][INFO][o.e.n.Node] [vAswxKt] starting ... [2019-06-26T14:07:21,898][INFO][o.e.t.TransportService] [vAswxKt] publish\_address {10.188.210.34:9300}, bound\_addresses {10.188.210.34:9300} [2019-06-26T14:07:22,344][WARN][o.e.b.BootstrapChecks] [vAswxKt] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2019-06-26T14:07:22,638][INFO][o.e.h.n.Netty4HttpServerTransport] [vAswxKt] publish\_address {10.188.210.34:9200}, bound\_addresses {10.188.210.34:9200} [2019-06-26T14:07:22,639][INFO][o.e.n.Node] [vAswxKt] started [2019-06-26T14:07:29,850][INFO][o.e.g.GatewayService] [vAswxKt] recovered [146] indices into cluster\_state [2019-06-26T14:08:31,203][DEBUG][o.e.a.s.TransportSearchAction] [vAswxKt] All shards failed for phase: [query] [2019-06-26T14:08:31,204][WARN][r.suppressed] path: /dmarc\_aggregate\*/\_search, params: {index=dmarc\_aggregate\*} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:274) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:132) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:243) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:205) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:184) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_192] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_192] [2019-06-26T14:08:32,227][DEBUG][o.e.a.s.TransportSearchAction] [vAswxKt] All shards failed for phase: [query] [2019-06-26T14:08:32,227][WARN][r.suppressed] path: /dmarc\_aggregate\*/\_search, params: {index=dmarc\_aggregate\*} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:274) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:132) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:243) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:205) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:184) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_192] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_192] [2019-06-26T14:08:35,244][DEBUG][o.e.a.s.TransportSearchAction] [vAswxKt] All shards failed for phase: [query] [2019-06-26T14:08:35,244][WARN][r.suppressed] path: /dmarc\_aggregate\*/\_search, params: {index=dmarc\_aggregate\*} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:274) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:132) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:243) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:205) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:184) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.2.2.jar:6.2.2] at ---
**Author:** ![Analyst](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@Analyst](https://discuss.elastic.co/u/Analyst)\ **Post date:** [June 26, 2019, 4:36pm UTC](https://discuss.elastic.co/t/all-shards-failing/187626/3 "2019-06-26T16:36:36Z")
GET /\_cat/indices gives the below output red open winlogbeat-6.6.1-2019.05.11 9Qejm9hXSGaZXuuDfOJstg 1 1 red open paloalto-web-logs xG0ah1N5QXaVAmIfHoz07g 5 1 red open nifi-stats uJeSz4z6SiWtgD-Yz\_r4jA 1 1 red open winlogbeat-6.6.1-2019.03.20 IdSGFA23Q9CIyCjt0uHMQw 1 1 red open winlogbeat-6.6.1-2019.05.09 oMolshskToaTK7sassDNwg 1 1 red open winlogbeat-6.6.1-2019.05.20 Ac36sHcxQxGbgddopghiuA 1 1 red open winlogbeat-6.6.1-2019.02.27 pUqXaRO-SSiaLoskkVcykw 1 1 red open dmarc\_aggregate\_dmarc-2019-06-13 dXGfUOpSRauwaRkalZJ1Gw 1 1 red open dmarc\_aggregate\_dmarc-2019-06-15 pD7IfdluRgmDPFbRqD7m9A 1 1 red open winlogbeat-6.6.1-2019.06.25 \_HfkDiKEQhiaESp-LLf7AQ 1 1 red open winlogbeat-6.6.1-2019.06.04 suJVgcrbTSmwqZyjstKJng 1 1 red open dmarc\_aggregate\_dmarc-2019-06-18 Jhi1TSQASkOhSfi\_f6gZvA 1 1 red open winlogbeat-6.6.1-2019.05.05 3dOdoe-WSSK-m0XdekZneg 1 1 red open winlogbeat-6.6.1-2019.04.21 2jEQynufSUmV8D9AJwAhQA 1 1 red open winlogbeat-6.6.1-2019.04.23 dJ6XBVwRQ9qrWhpHpAMBgw 1 1 red open winlogbeat-6.6.1-2019.05.03 W5Tdf-c8RS-vWiEfshrdig 1 1 red open winlogbeat-6.6.1-2019.06.19 UvAO\_HeMRgiAWqffsaNo8A 1 1 red open dmarc\_aggregate\_dmarc-2019-06-23 f5dq16YvStWb5ibPwnMXew 1 1 red open winlogbeat-6.6.1-2019.06.17 yg-US-cuTlm20RuAER4Y9Q 1 1 red open dmarc\_aggregate\_dmarc-2019-06-14 mzZadbM6SMivFNtIo9jqPA 1 1 red open filebeat-6.2.2-2019.02.14 YWHj0agDRvOC8Dj3Agm9hg 3 1 red open winlogbeat-6.6.1-2019.06.21 5HsumnRRRL-pIxg\_mms67g 1 1 red open winlogbeat-6.6.1-2019.03.24 qhpFpxAeR5-OaWVy1oNIMA 1 1 red open winlogbeat-6.6.1-2019.05.18 bLgtM1\_1RAG0ayaRXybyxA 1 1 red open filebeat-6.2.2-2019.02.19 nUb3C3LYSjWW59AXQ6Gqfw 3 1 red open winlogbeat-6.6.1-2019.04.16 iBKjN-e9QUq4CPpFvbJY5Q 1 1 red open winlogbeat-6.6.1-2019.06.15 SpunZ4GeT-ydYMhtjka69w 1 1 red open winlogbeat-6.6.1-2019.04.25 hLsp0sTCQv2mDliWlv0mXQ 1 1 red open filebeat-6.2.2-2019.02.28 6o6eH4YDRNClrDc5w4rgyg 3 1 yellow open test dE8-jxZRR9SN9rpsZuuyvQ 5 1 1 0 4.5kb 4.5kb red open filebeat-6.2.2-2019.02.24 ROpOR5onT7GRw48HybtDsQ 3 1 red open winlogbeat-6.6.1-2019.06.23 2Kdd1G91T6q2mY-dyItygA 1 1 red open winlogbeat-6.6.1-2019.04.30 OtFUx2rQS-CSNhBevIkbVg 1 1 red open filebeat-6.2.2-2019.02.21 aknCZbUTRSGZGTwEsE3Hkg 3 1 red open winlogbeat-6.6.1-2019.04.02 GoiMPPBuS7KK4Ii1yv3w8w 1 1 red open winlogbeat-6.6.1-2019.06.05 tKaoo7rNT\_WNyZkhx6kAPQ 1 1 red open filebeat-6.2.2-2019.03.04 kBowFA1uRISvicoZA1X-Gw 3 1 red open winlogbeat-6.6.1-2019.05.23 dCUJg\_UmQpu6dJoBDi\_ibw 1 1 red open winlogbeat-6.6.1-2019.05.22 HuS1wpFYRcyaytGm5wbm0Q 1 1 red open winlogbeat-6.6.1-2019.06.16 LO07cNWcSc-VFRsY3G2brA 1 1 red open nifi-queue-stats AjY8Qt8sRiSDVjs3DcrzhQ 2 1 green open json ABWhJBF-S2CplUBP\_dWXSw 5 0 0 0 1.2kb 1.2kb red open winlogbeat-6.6.1-2019.06.06 sPLxwRDhR3KDi6jKYbnVJg 1 1 red open winlogbeat-6.6.1-2019.04.15 uI7jqLH\_QFa4w32CHT9Jzw 1 1 red open winlogbeat-6.6.1-2019.05.31 cyL-MzCiREaIgQsrpAG55A 1 1 red open filebeat-6.2.2-2019.02.22 DU5EFtpITHOEQiBuspVkpA 3 1 red open winlogbeat-6.6.1-2019.06.08 tQr2ahPHTxWCL29mnuARVA 1 1 red open winlogbeat-6.6.1-2019.05.15 c8ixDbD9RfG6pBdktNCSBQ 1 1 red open winlogbeat-6.6.1-2019.03.26 mH9AVy0qQiWaDyiwqyXjUA 1 1 red open winlogbeat-6.6.1-2019.05.13 E527uSu-RSaZ7Kr1U-tCMQ 1 1 red open winlogbeat-6.6.1-2019.06.07 oUY75v-ARB6wPBeacHtOSw 1 1 red open winlogbeat-6.6.1-2019.06.22 aD7VHeZJTLCec2iRfIZP4A 1 1 red open winlogbeat-6.6.1-2019.06.09 wqLzV\_uuTLiYANzg7tsOCA 1 1 red open winlogbeat-6.6.1-2019.05.25 AlHOVIcNTv64hwQ9mvXDZA 1 1 red open winlogbeat-6.6.1-2019.03.22 D5teR5KbRKq6I0W4Ez6pXQ 1 1 red open filebeat-6.2.2-2019.02.25 SmuTHwhiQkyyuwEMvILuVA 3 1 red open filebeat-6.2.2-2019.02.16 JP3Lz3swRheET2smuC3jbQ 3 1 red open winlogbeat-6.6.1-2019.03.28 Kp4ib0SNSPmdfmhnuZ9ouA 1 1 red open filebeat-6.2.2-2019.03.03 pP9V-60\_S4aV716nV2ETbw 3 1 red open winlogbeat-6.6.1-2019.05.19 b\_xycYagROavxmbdG0s2og 1 1 red open winlogbeat-6.6.1-2019.05.16 siJARfeSRZWF037kLBaUcA 1 1 red open winlogbeat-6.6.1-2019.05.01 jHLcRq5CRcafRkbCjAVmMg 1 1 red open dmarc\_aggregate\_dmarc-2019-06-12 1\_9YTVpyRYCAEHzJ-vM9HQ 1 1 red open filebeat-6.2.2-2019.02.11 aGjQto07RJK\_NNA1ls\_49A 3 1 red open filebeat-6.2.2-2019.02.27 vSTd-e9uS\_KIqmkOqzZitw 3 1 red open winlogbeat-6.6.1-2019.04.19 TrvMBPtKSPKpFVGBDF\_OVA 1 1 red open winlogbeat-6.6.1-2019.03.16 iFxyfZiBSpqVfPKn4VLkDw 1 1 red open winlogbeat-6.6.1-2019.03.15 -6WfRalbQBCk-HY-iARjlA 1 1 * * * ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 27, 2019, 7:57am UTC](https://discuss.elastic.co/t/all-shards-failing/187626/4 "2019-06-27T07:57:57Z")
Hey, searching against the `dmarc_aggregate` indices failed. Can you share a full stack trace from the client side? See [https://www.elastic.co/guide/en/elasticsearch/reference/7.2/common-options.html#common-options-error-options](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/common-options.html#common-options-error-options) Thanks! --Alex ---
**Author:** ![Analyst](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@Analyst](https://discuss.elastic.co/u/Analyst)\ **Post date:** [June 27, 2019, 9:27pm UTC](https://discuss.elastic.co/t/all-shards-failing/187626/5 "2019-06-27T21:27:35Z")
Hi Alexander, Thanks a lot for your response. I was very impatient so i removed elasticserch and reconfigured it. That solved the problem. Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 25, 2019, 9:27pm UTC](https://discuss.elastic.co/t/all-shards-failing/187626/6 "2019-07-25T21:27:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.