org.elasticsearch.action.UnavailableShardsException: [ . async—searchl [01 primary shard is not active Timeout: [1m

[forabraham1 (Abraham)]

After upgrading ES from 7.9 to 7.17 ES is throwing this exception "org.elasticsearch.action.UnavailableShardsException: [ . async—searchl [01 primary shard is not
active Timeout: [1m". Due to which Kibana isn't returning any result and just hung.
Our index storage is kept in external disk. Before doing the update VM snapshots were taken (excluding the index storage drive). Upgrade was rolled back (by reverting VM snapshots) as we had issue in kibana upgrade. Again we upgraded to 7.17 and this time all came up but, ES started to throw this exception and due to this Kibana is hung.

Any lead to fix this would be greatly appreciated.

Thank you

[warkolm (Mark Walkom)]

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

[forabraham1 (Abraham)]

Got it. will take care in future posts. Pasting the Exception trace.

[WARN ] [o.e.x.m.e.I.LocaIExporter] unexpected error while indexing monitoring document
elasticcsearch.xpack.monltoring. exporter.ExportException:UnavailableShardsException[ [.monitoringkibana-7-2023.01.19] [O] primary shard is not active Timeout : [1m], request: [BulkShardException]
org.elastlcsearch.xpack.monltoring.exporter.local.LocaIBuIk.lambda$throwExportException$2 (Local Bulk. java: 135) .17 . 6. jar

.xpack. security. action. filter. SecurityActionFiIter.Iambda$appIyInternaI$2 (SecurityActionFiIter.iava:IS2) .17 .6.iar:7.17 . 6]
. e I as e arch
.ActionListener$DeIegatingFaiIureActionListener. onResponse (ActionListener.iava:21S) [elastIcsearch—7 .17 . 6 . jar
. elastlcsearch. act Ion
:7.17.6]
.ActionListener$RunBeforeActionListener. onResponse 389) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6]
. elastlcsearch. act Ion
. ActionListener$MappecActionListener. onResponse (ActionListener.iava:IOI) [elastIcsearch—7 .17 . 6 . jar
. elastlcsearch. act Ion
:7.17.6]
.bulk. TransportBuIkÄction$BuIkOperation$I . 625) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6]
. elastlcsearch . act Ion
.BuIk. TransportBuIkAction$BuIkOperation$I . onFaiIure (TransportBuIkÄction.iava: 620) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6]
. elastlcsearch. act Ion
. support . TransportAction$I . onFaiIure ( TransportAction . ava : 97 )
[elastIcsearch—7 .17 . 6 . jar
. elastlcsearch. act Ion
:7.17.6]
. support .ContextpreservingActionLIstener. onFaiIure (ContextpreservingActionLIstener.iava:38) [elasticsearch—7 .17 . 6. jar:7 .17 . 6]
. elastlcsearch. act Ion
. ActionListener$DeIegating. onFaiIure (ActionListener.iava: 66) [elastIcsearch—7 .17 . 6 . jar: 7 .17 . 6]
. elastlcsearch . act Ion
. support . (TransportRepIicationÄction.iava:1041) [elasticsearch—7 .17 . 6]
. elastlcsearch. act Ion
. support . . retry (TransportRepIicationÄction.iava:1013) [elastIcsearch—7 .17 . 6. iar:7.17 .6]
. elastlcsearch. act Ion
. support . replication . TransportRepIicatIonAction$Reroutephase . retryBecauseUnavaiIaBIe ( TransportRepIicatIonAction . ava : 1077 )
[elasticsearch—7 .17 . 6 . jar: 7 .17 . 6]
. elastlcsearch. act Ion
. e last csearch . act on
. e last I csearch . common . ut
. elastlcsearch. act Ion
. elastlcsearch
. elastlcsearch
. csearch
. elastlcsearch . common . ut
. util . stream. ReferencepipeIine$3$I . accept (Referencepipeline . java : 197 )
. util . stream. ReferencepipeIine$2$I . accept (Referencepipeline . ava : 179)
. Spli terator . forEachRemaIning ( terators . ava : 992)
. util . stream. Abstractpipeline . copy Into (Ahstractpipeline . ava : SOS)
. util . stream. Abstractpipeline . wrapAndCopyInto ( Abstractpipeline . java : 4 SS )
. util . stream. ForEachOps$ForEachOp . evaluatesequential ( ForEachOps . ava : 150)
. util . stream. ForEachOps$ForEachOp$OfRef . evaluatesequential ( ForEachOps . ava : 173)
. util . stream. Abstractpipeline . evaluate ( Abstractpipeline . i ava : 234)
. util . stream. Referencepipeline . forEach (Referencepipeline . ava : 596)
.xpack.monltoring. exporter. local .LocaIBuIk. throwExportException (Local Bulk. java: 136) .17 . 6 . jar: 7 .17 . 6]
.xpack.monitoring. exporter. local .LocaIBuIk. lambda$doFIush$O (Local Bulk. java: 117) .17 . 6 . jar: 7 .17 . 6]
. actIon.ActionListener$I . onResponse (ActionListener.iava:136) [elastIcsearch—7 .17 . 6 . jar: 7 .17 . 6]
. act Ion. support .ContextpreservingActionLIstener. onResponse (ContextpreservingActionLIstener.iava:31) [elasticsearch—7 .17 .17 . 6]
. act Ion. support . TransportAction$I . onResponse (TransportAction. fava: 88) [elastIcsearch—7 .17 . 6 . jar
:7.17.6]
. act Ion. support . TransportAction$I . onResponse (TransportAction. fava: 82) [elastIcsearch—7 .17 . 6 . jar
:7.17.6]
. act Ion. support .ContextpreservingActionLIstener. onResponse (ContextpreservingActionLIstener.iava:31) [elasticsearch—7 .17 .17 . 6]
. support . replication. TransportRePreformatted textpIicationÄction$Reroutephase.doRun (TransportRepIicationÄction.iava:873) [elastIcsearch—7 .17 . 6]
. concurrent . AhstractRunnaBIe . run (AhstractRunnaBIe . 26) [elastIcsearch—7 .17 .
:7.17.6]
. support . replication. TransportRepIicationÄction$Reroutephase$2.onTimeout (TransportRepIicationÄction.iava:1032) [elasticsearch—7 .17 . 6. jar:7
. cluster. ClusterStateObserver$ContextpreservingListener. onTimeout (ClusterStateOBserver.iava:345) [elastIcsearch—7 .17 . 6. jar
:7.17.6]
. cluster . ClusterStateOBserver$OBserverCIusterStateListener . imeout ( ClusterStateOBserver . ava : 263)
[elastIcsearch—7 .17 . 6. far
:7.17.6]
. cluster. service 660) [elasticsearch—7 .17 . 6. iar:7 .17 . 6]
. concurrent . ThreadContext$ContextpreservingRunnaBIe . run ( ThreadContext . ava
[elasticsearch—7 .17 . 6 . jar: 7 .17 . 6]
: 718)
. concurrent . ThreadPooIExecutor . runWorker ( ThreadPooIExecutor . ava : 1136)
java . ut
. concurrent . ThreadpooIExecutor$Worker . run (ThreadPooIExecutor . java : 635 )
java. lang . Thread. run (Thread. i ava: 833) [ ? : ? ]
org . e I as t I csearch . act I on . Unavai I abl e ShardsExcepti on :
O]
p r Imary
shard
T Ime out :
request :
[ Bu I k Sha s t
.17.6]
. t or Ing —
k bana
-7-2023.01.19]
cont a:

[forabraham1 (Abraham)]

One more log trace

[WARN ] [r.suppressed ] path: async search, params: [ignore unavailable=true, preference=16740S565SS57, index=XXX_logs—* keep on completion=true, keep alive=604800000ms, batched reduce slze=64, wait for completion timeout=IOOms, enable fields emulation=true, track total hits=true} 
org . eIasticsearch.action.UnavailableShardsException: . async—search] [O] primary shard is not active Timeout: [1m], request: [BulkShardRequest . async—search] [O] ] 
containing [index { . async—search] doc] , source [l "headers": ["_xpack security authentication": "i SWIAwAHZWxhc3RpYwEJc3VwZXJ1 c2VycgEJX3 Jl as 11bGOwMDEt cmV z ZXJ2ZWQ1 
cmVzZXJ2ZWQAAAoA" , " expiration_time " : 1674707861559 , " : " i SWIAORGTABi SHDLDXNKCXEr8€vKgQoN8Yu1cosKT zHMS1wOdi QNDnVNSQ03NAgOMHVLdE8Ji Yw1Mwz0DQoNrM02Spc1y06pyg/wyOgv8AOJtGVgzpx1kpmh/ gADGDAy8gUk1gTGZ2QW1 +snFyXlaMhrSZ2GEyDEwc1L1si Jz  " result 
USWLs40ykOviwQL5BSWZ+yxnJOWCZmprUgvzki p: czJyczGJe1 SKi/1JUfZ/8vJQ8PJAZ8kBLWk84rRT05AHZ13rCWc5hpgCUob1gHi +E€cKxYKow1F17ysdMSDPDUag21wyDe1BtrncTTg1c1LoFTfi GDWUepzAHgsDd5sM3Toio+yaF2vcIHuoBzUIQUY85Ddohai OFN/ zig4hC-,ti 5SX+4cpwxp6vcb6ht/ IUBW2Gi 7xu2QwOLSNmwhR20iYmBUR4YtkDAIQeJ 
ytYTgT>m1ppNHIM5AAAÄ/ / 8 DAA= " }]}]
at org.elasticsearch.action.support. replication.TransportRepIicationÄction$Reroutephase . retryBecauseUnavaiIabIe (TransportRepIicationÄction . ava : 1077 ) [elasticsearch—7 .17 . 6 . jar: 7 .17 . 6] 
at org.elasticsearch.action. support . replication. TransportRepIicationAction$Reroutephase.doRun (TransportRepIicationÄction.iava:873) [elasticsearch—7 .17 . 6]

[forabraham1 (Abraham)]

One more log trace.

[2023-01-19T00:00:02,544][WARN ][r.suppressed             ] [xxxxx] path: /xxx-*/_async_search, params: {ignore_unavailable=true, index=xxxx-*, keep_on_completion=true, keep_alive=604800000ms, batched_reduce_size=64, wait_for_completion_timeout=100ms, enable_fields_emulation=true, track_total_hits=true}
org.elasticsearch.action.UnavailableShardsException: [.async-search][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.async-search][0]] containing [index {[.async-search][_doc][2_lisz2cT3i6en7A3Pz8uA], source[n/a, actual length: [2.9kb], max length: 2kb]}]]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1077) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:873) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1032) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.6.jar:7.17.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-01-19T00:00:02,544][WARN ][r.suppressed             ] [xxxxx] path: /xxxxx-*/_async_search, params: {ignore_unavailable=true, index=xxxx-*, keep_on_completion=true, keep_alive=604800000ms, batched_reduce_size=64, wait_for_completion_timeout=100ms, enable_fields_emulation=true, track_total_hits=true}
org.elasticsearch.action.UnavailableShardsException: [.async-search][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.async-search][0]] containing [index {[.async-search][_doc][6_ibY0P7RuCTUbbhUF0Ugg], source[{"headers":{"_xpack_security_authentication":"i9W1AwAHbW9uaXRvcgIPbW9uaXRvcmluZ191c2VyB21vbml0b3IKAAEKTW9uaXRvcmluZwEAAQANbG1pLWVsbTAwMS1wMQ5kZWZhdWx0X25hdGl2ZQZuYXRpdmUAAAoA"},"expiration_time":1674691142453,"result":"i9W1A0RGTABi9HDLikpLDPeK8nVzNA7NC3MJcwvLjMzNCAvKcgqLMkpOTQo3NAg0MHVLdE8JjYwIMwzODQoNrMo29PcIy06pyq/wd0k28K3yNWZgZGRgqD/AAAaMjGzFJalFucUqZobJiabmaYa6JmaWKbqGhqnmuolpBka6ZpapJuYGiWnmJobmXCzsBYl5qTmeKQww9Qb41PNl5gENL0vMCS4pysxLZ2A2MyjmKE4tykwtRjICr5WcJZm5qW6ZqTkpDFypZal5JfEggf9MjCwgzAX0C3NRYrkk2DNMDIzy8mAWmxzEd63Hg+JNQfTruMemAAAAAP//AwA="}]}]]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1077) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:873) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1032) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.6.jar:7.17.6]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.6.jar:7.17.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]