As for a week an index has red health and is unresponsive after es startup.
Triggering a refresh, clear cache or reindex makes the index responsive again but health is still red.
Index status
red open index-name iDbqqw_7TCS9QPoUiN0XDg 1 1 523356487 0 135.6gb 135.6gb
Elasticsearch Log:
[2020-01-23T10:08:28,279][DEBUG][o.e.a.s.TransportSearchAction] [DESKTOP-Q7V5E0R] All shards failed for phase: [query] [2020-01-23T10:08:28,325][WARN ][r.suppressed ] [DESKTOP-Q7V5E0R] path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:305) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1223) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1197) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:835) [?:?] [2020-01-23T10:39:12,309][INFO ][o.e.c.r.a.AllocationService] [DESKTOP-Q7V5E0R] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[vehicles-scooter-tier-snapshot][0]] ...]). [2020-01-23T14:22:32,408][WARN ][o.e.c.s.MasterService ] [DESKTOP-Q7V5E0R] cluster state update task [shard-started StartedShardEntry{shardId [[index-name][0]], allocationId [ObzDmaQgQBWDRLeg_AZ_2w], primary term [21], message [after existing store recovery; bootstrap_history_uuid=false]}[StartedShardEntry{shardId [[index-name][0]], allocationId [ObzDmaQgQBWDRLeg_AZ_2w], primary term [21], message [after existing store recovery; bootstrap_history_uuid=false]}]] took [3.7h] which is above the warn threshold of 30