# Helping with red shards **URL:** \ **Category:** Elasticsearch\ **Created:** [July 18, 2020, 3:57pm UTC](https://discuss.elastic.co/t/helping-with-red-shards/241748 "2020-07-18T15:57:58Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![AayushPatel](https://avatars.discourse-cdn.com/v4/letter/a/da6949/32.png) [@AayushPatel](https://discuss.elastic.co/u/AayushPatel)\ **Post date:** [July 18, 2020, 3:57pm UTC](https://discuss.elastic.co/t/helping-with-red-shards/241748/1 "2020-07-18T15:57:58Z")
Hi I have elastic on 1 node and tried to reindex but it maked my node health red and after i delete the destination index to free space all of my shards get red and do not get back to run elastic the data is important to me and can not delete indexes ```auto [2018-03-05T05:56:29,826][DEBUG][o.e.a.s.TransportSearchAction] [93CvEwd] All shards failed for phase: [query] [2018-03-05T05:56:29,828][DEBUG][o.e.a.s.TransportSearchAction] [93CvEwd] All shards failed for phase: [query] [2018-03-05T05:56:29,829][DEBUG][o.e.a.s.TransportSearchAction] [93CvEwd] All shards failed for phase: [query] [2018-03-05T05:56:32,924][WARN][o.e.m.j.JvmGcMonitorService] [93CvEwd] [gc][young][2538][518] duration [1.5s], collections [1]/[1.8s], total [1.5s]/[52s], memory [10.7gb]->[9.8gb]/[21.8gb], all_pools {[young] [1011.8mb]->[11.1mb]/[1gb]}{[survivor] [121.3mb]->[127.7mb]/[133mb]}{[old] [9.6gb]->[9.7gb]/[20.7gb]} [2018-03-05T05:56:32,924][WARN][o.e.m.j.JvmGcMonitorService] [93CvEwd] [gc][2538] overhead, spent [1.5s] collecting in the last [1.8s] [2018-03-05T05:56:38,865][DEBUG][o.e.a.s.TransportSearchAction] [93CvEwd] All shards failed for phase: [query] [2018-03-05T05:56:38,865][WARN][r.suppressed] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:210) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:189) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] [2018-03-05T06:10:43,660][WARN][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-es-6-2018.03.05][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-6-2018.03.05][0]] containing [10509] requests]] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_171] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_171] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_171] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_171] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_171] at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_171] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_171] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_171] at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_171] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:129) ~[?:?] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:111) ~[?:?] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:570) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:379) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:374) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:897) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:869) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:928) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryIfUnavailable(TransportReplicationAction.java:774) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:727) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:888) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:576) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] Caused by: org.elasticsearch.action.UnavailableShardsException: [.monitoring-es-6-2018.03.05][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-6-2018.03.05][0]] containing [10509] requests] ... 12 more ``` this is elasticsearch logs, I tested the number of shards and get 10507 total shards 6489 are UNASSIGNED then i tried, ```auto curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_cluster/settings' -d '{ "persistent" : {"cluster.max_shards_per_node" : 5000'}} ``` but i get ```auto {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"persistent setting [cluster.max_shards_per_node], not recognized"}],"type":"illegal_argument_exception","reason":"persistent setting [cluster.max_shards_per_node], not recognized"},"status":400} ``` I done everything i can except delete indices any help would be grateful tnx ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 21, 2020, 12:09am UTC](https://discuss.elastic.co/t/helping-with-red-shards/241748/2 "2020-07-21T00:09:19Z")
You have way too many shards for a single node. We recommend running \<700 a node. > [@AayushPatel](#): > > I done everything i can except delete indices It looks like you have tonnes of old data (eg `.monitoring-es-6-2018.03.05`), you should remove these. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 18, 2020, 12:09am UTC](https://discuss.elastic.co/t/helping-with-red-shards/241748/3 "2020-08-18T00:09:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.