# org.elasticsearch.action.UnavailableShardsException: \[ . async—searchl \[01 primary shard is not active Timeout: \[1m **URL:** \ **Category:** Elasticsearch\ **Created:** [January 19, 2023, 3:37am UTC](https://discuss.elastic.co/t/org-elasticsearch-action-unavailableshardsexception-async-searchl-01-primary-shard-is-not-active-timeout-1m/323466 "2023-01-19T03:37:54Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![forabraham1](https://avatars.discourse-cdn.com/v4/letter/f/7feea3/32.png) [@forabraham1](https://discuss.elastic.co/u/forabraham1)\ **Post date:** [January 19, 2023, 3:37am UTC](https://discuss.elastic.co/t/org-elasticsearch-action-unavailableshardsexception-async-searchl-01-primary-shard-is-not-active-timeout-1m/323466/1 "2023-01-19T03:37:54Z")
After upgrading ES from 7.9 to 7.17 ES is throwing this exception "org.elasticsearch.action.UnavailableShardsException: [ . async—searchl [01 primary shard is not active Timeout: [1m". Due to which Kibana isn't returning any result and just hung. Our index storage is kept in external disk. Before doing the update VM snapshots were taken (excluding the index storage drive). Upgrade was rolled back (by reverting VM snapshots) as we had issue in kibana upgrade. Again we upgraded to 7.17 and this time all came up but, ES started to throw this exception and due to this Kibana is hung. Any lead to fix this would be greatly appreciated. Thank you ![Kibana Error](https://us1.discourse-cdn.com/elastic/original/3X/5/2/5269b8cf994dcaf2d4587ec7a789330cd4ea0564.png) ![Logs](https://us1.discourse-cdn.com/elastic/original/3X/1/1/116ece2d2c8d7f42a1d722915f030d5bc8554f00.png) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 19, 2023, 3:41am UTC](https://discuss.elastic.co/t/org-elasticsearch-action-unavailableshardsexception-async-searchl-01-primary-shard-is-not-active-timeout-1m/323466/2 "2023-01-19T03:41:37Z")
Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them 🙂 ---
**Author:** ![forabraham1](https://avatars.discourse-cdn.com/v4/letter/f/7feea3/32.png) [@forabraham1](https://discuss.elastic.co/u/forabraham1)\ **Post date:** [January 19, 2023, 4:35am UTC](https://discuss.elastic.co/t/org-elasticsearch-action-unavailableshardsexception-async-searchl-01-primary-shard-is-not-active-timeout-1m/323466/3 "2023-01-19T04:35:50Z")
Got it. will take care in future posts. Pasting the Exception trace. [WARN] [o.e.x.m.e.I.LocaIExporter] unexpected error while indexing monitoring document elasticcsearch.xpack.monltoring. exporter.ExportException:UnavailableShardsException[[.monitoringkibana-7-2023.01.19] [O] primary shard is not active Timeout : [1m], request: [BulkShardException] org.elastlcsearch.xpack.monltoring.exporter.local.LocaIBuIk.lambda$throwExportException$2 (Local Bulk. java: 135) .17 . 6. jar .xpack. security. action. filter. SecurityActionFiIter.Iambda$appIyInternaI$2 (SecurityActionFiIter.iava:IS2) .17 .6.iar:7.17 . 6] . e I as e arch .ActionListener$DeIegatingFaiIureActionListener. onResponse (ActionListener.iava:21S) [elastIcsearch—7 .17 . 6 . jar . elastlcsearch. act Ion :7.17.6] .ActionListener$RunBeforeActionListener. onResponse 389) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6] . elastlcsearch. act Ion . ActionListener$MappecActionListener. onResponse (ActionListener.iava:IOI) [elastIcsearch—7 .17 . 6 . jar . elastlcsearch. act Ion :7.17.6] .bulk. TransportBuIkÄction$BuIkOperation$I . 625) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6] . elastlcsearch . act Ion .BuIk. TransportBuIkAction$BuIkOperation$I . onFaiIure (TransportBuIkÄction.iava: 620) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6] . elastlcsearch. act Ion . support . TransportAction$I . onFaiIure ( TransportAction . ava : 97 ) [elastIcsearch—7 .17 . 6 . jar . elastlcsearch. act Ion :7.17.6] . support .ContextpreservingActionLIstener. onFaiIure (ContextpreservingActionLIstener.iava:38) [elasticsearch—7 .17 . 6. jar:7 .17 . 6] . elastlcsearch. act Ion . ActionListener$DeIegating. onFaiIure (ActionListener.iava: 66) [elastIcsearch—7 .17 . 6 . jar: 7 .17 . 6] . elastlcsearch . act Ion . support . (TransportRepIicationÄction.iava:1041) [elasticsearch—7 .17 . 6] . elastlcsearch. act Ion . support . . retry (TransportRepIicationÄction.iava:1013) [elastIcsearch—7 .17 . 6. iar:7.17 .6] . elastlcsearch. act Ion . support . replication . TransportRepIicatIonAction$Reroutephase . retryBecauseUnavaiIaBIe ( TransportRepIicatIonAction . ava : 1077 ) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6] . elastlcsearch. act Ion . e last csearch . act on . e last I csearch . common . ut . elastlcsearch. act Ion . elastlcsearch . elastlcsearch . csearch . elastlcsearch . common . ut . util . stream. ReferencepipeIine$3$I . accept (Referencepipeline . java : 197 ) . util . stream. ReferencepipeIine$2$I . accept (Referencepipeline . ava : 179) . Spli terator . forEachRemaIning ( terators . ava : 992) . util . stream. Abstractpipeline . copy Into (Ahstractpipeline . ava : SOS) . util . stream. Abstractpipeline . wrapAndCopyInto ( Abstractpipeline . java : 4 SS ) . util . stream. ForEachOps$ForEachOp . evaluatesequential ( ForEachOps . ava : 150) . util . stream. ForEachOps$ForEachOp$OfRef . evaluatesequential ( ForEachOps . ava : 173) . util . stream. Abstractpipeline . evaluate ( Abstractpipeline . i ava : 234) . util . stream. Referencepipeline . forEach (Referencepipeline . ava : 596) .xpack.monltoring. exporter. local .LocaIBuIk. throwExportException (Local Bulk. java: 136) .17 . 6 . jar: 7 .17 . 6] .xpack.monitoring. exporter. local .LocaIBuIk. lambda$doFIush$O (Local Bulk. java: 117) .17 . 6 . jar: 7 .17 . 6] . actIon.ActionListener$I . onResponse (ActionListener.iava:136) [elastIcsearch—7 .17 . 6 . jar: 7 .17 . 6] . act Ion. support .ContextpreservingActionLIstener. onResponse (ContextpreservingActionLIstener.iava:31) [elasticsearch—7 .17 .17 . 6] . act Ion. support . TransportAction$I . onResponse (TransportAction. fava: 88) [elastIcsearch—7 .17 . 6 . jar :7.17.6] . act Ion. support . TransportAction$I . onResponse (TransportAction. fava: 82) [elastIcsearch—7 .17 . 6 . jar :7.17.6] . act Ion. support .ContextpreservingActionLIstener. onResponse (ContextpreservingActionLIstener.iava:31) [elasticsearch—7 .17 .17 . 6] . support . replication. TransportRe`Preformatted text`pIicationÄction$Reroutephase.doRun (TransportRepIicationÄction.iava:873) [elastIcsearch—7 .17 . 6] . concurrent . AhstractRunnaBIe . run (AhstractRunnaBIe . 26) [elastIcsearch—7 .17 . :7.17.6] . support . replication. TransportRepIicationÄction$Reroutephase$2.onTimeout (TransportRepIicationÄction.iava:1032) [elasticsearch—7 .17 . 6. jar:7 . cluster. ClusterStateObserver$ContextpreservingListener. onTimeout (ClusterStateOBserver.iava:345) [elastIcsearch—7 .17 . 6. jar :7.17.6] . cluster . ClusterStateOBserver$OBserverCIusterStateListener . imeout ( ClusterStateOBserver . ava : 263) [elastIcsearch—7 .17 . 6. far :7.17.6] . cluster. service 660) [elasticsearch—7 .17 . 6. iar:7 .17 . 6] . concurrent . ThreadContext$ContextpreservingRunnaBIe . run ( ThreadContext . ava [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6] : 718) . concurrent . ThreadPooIExecutor . runWorker ( ThreadPooIExecutor . ava : 1136) java . ut . concurrent . ThreadpooIExecutor$Worker . run (ThreadPooIExecutor . java : 635 ) java. lang . Thread. run (Thread. i ava: 833) [? : ?] org . e I as t I csearch . act I on . Unavai I abl e ShardsExcepti on : O] p r Imary shard T Ime out : request : [ Bu I k Sha s t .17.6] . t or Ing — k bana -7-2023.01.19] cont a: ---
**Author:** ![forabraham1](https://avatars.discourse-cdn.com/v4/letter/f/7feea3/32.png) [@forabraham1](https://discuss.elastic.co/u/forabraham1)\ **Post date:** [January 19, 2023, 4:47am UTC](https://discuss.elastic.co/t/org-elasticsearch-action-unavailableshardsexception-async-searchl-01-primary-shard-is-not-active-timeout-1m/323466/4 "2023-01-19T04:47:13Z")
One more log trace ```auto [WARN] [r.suppressed] path: async search, params: [ignore unavailable=true, preference=16740S565SS57, index=XXX_logs—* keep on completion=true, keep alive=604800000ms, batched reduce slze=64, wait for completion timeout=IOOms, enable fields emulation=true, track total hits=true} org . eIasticsearch.action.UnavailableShardsException: . async—search] [O] primary shard is not active Timeout: [1m], request: [BulkShardRequest . async—search] [O] ] containing [index { . async—search] doc] , source [l "headers": ["_xpack security authentication": "i SWIAwAHZWxhc3RpYwEJc3VwZXJ1 c2VycgEJX3 Jl as 11bGOwMDEt cmV z ZXJ2ZWQ1 cmVzZXJ2ZWQAAAoA" , " expiration_time " : 1674707861559 , " : " i SWIAORGTABi SHDLDXNKCXEr8€vKgQoN8Yu1cosKT zHMS1wOdi QNDnVNSQ03NAgOMHVLdE8Ji Yw1Mwz0DQoNrM02Spc1y06pyg/wyOgv8AOJtGVgzpx1kpmh/ gADGDAy8gUk1gTGZ2QW1 +snFyXlaMhrSZ2GEyDEwc1L1si Jz " result USWLs40ykOviwQL5BSWZ+yxnJOWCZmprUgvzki p: czJyczGJe1 SKi/1JUfZ/8vJQ8PJAZ8kBLWk84rRT05AHZ13rCWc5hpgCUob1gHi +E€cKxYKow1F17ysdMSDPDUag21wyDe1BtrncTTg1c1LoFTfi GDWUepzAHgsDd5sM3Toio+yaF2vcIHuoBzUIQUY85Ddohai OFN/ zig4hC-,ti 5SX+4cpwxp6vcb6ht/ IUBW2Gi 7xu2QwOLSNmwhR20iYmBUR4YtkDAIQeJ ytYTgT>m1ppNHIM5AAAÄ/ / 8 DAA= " }]}] at org.elasticsearch.action.support. replication.TransportRepIicationÄction$Reroutephase . retryBecauseUnavaiIabIe (TransportRepIicationÄction . ava : 1077 ) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6] at org.elasticsearch.action. support . replication. TransportRepIicationAction$Reroutephase.doRun (TransportRepIicationÄction.iava:873) [elasticsearch—7 .17 . 6] ``` ---
**Author:** ![forabraham1](https://avatars.discourse-cdn.com/v4/letter/f/7feea3/32.png) [@forabraham1](https://discuss.elastic.co/u/forabraham1)\ **Post date:** [January 19, 2023, 11:17am UTC](https://discuss.elastic.co/t/org-elasticsearch-action-unavailableshardsexception-async-searchl-01-primary-shard-is-not-active-timeout-1m/323466/6 "2023-01-19T11:17:19Z")
One more log trace. ```auto [2023-01-19T00:00:02,544][WARN][r.suppressed] [xxxxx] path: /xxx-*/_async_search, params: {ignore_unavailable=true, index=xxxx-*, keep_on_completion=true, keep_alive=604800000ms, batched_reduce_size=64, wait_for_completion_timeout=100ms, enable_fields_emulation=true, track_total_hits=true} org.elasticsearch.action.UnavailableShardsException: [.async-search][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.async-search][0]] containing [index {[.async-search][_doc][2_lisz2cT3i6en7A3Pz8uA], source[n/a, actual length: [2.9kb], max length: 2kb]}]] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1077) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:873) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1032) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.6.jar:7.17.6] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] [2023-01-19T00:00:02,544][WARN][r.suppressed] [xxxxx] path: /xxxxx-*/_async_search, params: {ignore_unavailable=true, index=xxxx-*, keep_on_completion=true, keep_alive=604800000ms, batched_reduce_size=64, wait_for_completion_timeout=100ms, enable_fields_emulation=true, track_total_hits=true} org.elasticsearch.action.UnavailableShardsException: [.async-search][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.async-search][0]] containing [index {[.async-search][_doc][6_ibY0P7RuCTUbbhUF0Ugg], source[{"headers":{"_xpack_security_authentication":"i9W1AwAHbW9uaXRvcgIPbW9uaXRvcmluZ191c2VyB21vbml0b3IKAAEKTW9uaXRvcmluZwEAAQANbG1pLWVsbTAwMS1wMQ5kZWZhdWx0X25hdGl2ZQZuYXRpdmUAAAoA"},"expiration_time":1674691142453,"result":"i9W1A0RGTABi9HDLikpLDPeK8nVzNA7NC3MJcwvLjMzNCAvKcgqLMkpOTQo3NAg0MHVLdE8JjYwIMwzODQoNrMo29PcIy06pyq/wd0k28K3yNWZgZGRgqD/AAAaMjGzFJalFucUqZobJiabmaYa6JmaWKbqGhqnmuolpBka6ZpapJuYGiWnmJobmXCzsBYl5qTmeKQww9Qb41PNl5gENL0vMCS4pysxLZ2A2MyjmKE4tykwtRjICr5WcJZm5qW6ZqTkpDFypZal5JfEggf9MjCwgzAX0C3NRYrkk2DNMDIzy8mAWmxzEd63Hg+JNQfTruMemAAAAAP//AwA="}]}]] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1077) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:873) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1032) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.6.jar:7.17.6] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.6.jar:7.17.6] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 16, 2023, 11:17am UTC](https://discuss.elastic.co/t/org-elasticsearch-action-unavailableshardsexception-async-searchl-01-primary-shard-is-not-active-timeout-1m/323466/7 "2023-02-16T11:17:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.