Elastic Search Startup Warnings

Elastic Stack Elasticsearch
1

Hi there all,

We are running into a massive number of Warnings when starting up ElasticSearch, these are all in elasticsearch.log and are very strange as they seem to no inhibit start up at all.

[2019-09-05T13:12:18,799][DEBUG][o.e.a.s.TransportSearchAction] [patronus] All shards failed for phase: [query] [2019-09-05T13:12:18,804][WARN ][r.suppressed ] [patronus] path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:305) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:835) [?:?] [2019-09-05T13:12:18,855][DEBUG][o.e.a.s.TransportSearchAction] [patronus] All shards failed for phase: [query] [2019-09-05T13:12:18,871][WARN ][r.suppressed ] [patronus] path: /.kibana/_search, params: {index=.kibana, filter_path=hits.hits._id,hits.hits._source} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:305) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:835) [?:?] [2019-09-05T13:12:18,889][DEBUG][o.e.a.s.TransportSearchAction] [patronus] All shards failed for phase: [query] [2019-09-05T13:12:18,890][WARN ][r.suppressed ] [patronus] path: /.kibana/_search, params: {index=.kibana} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:305) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:835) [?:?]

There are quite a few of these issue but then at the end:

[2019-09-05T13:12:27,654][INFO ][o.e.c.r.a.AllocationService] [patronus] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_task_manager][0]] ...]).

Are these warning something to worry about? As I have been unsuccessful in Googling any form of fix for them.
Thanks for the assistance!

2

Is there any application that try to access elasticsearch when it's in retarting mode ?
if yes, those are normal errors, when elsticsearch is in retard mode, it attemps to recover all shards, so if any application (kibana, logstash, ...) try to access (query API, index API ...) to that share es will throws thoses exceptions ...

1 Like
3

That is very much the case. As we will have Logstash and Filebeats, and possibly Kibana trying to access it while it is restoring Shards.
Thank you!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.