Elasticsearch WARN log : "message": "path: /.kibana_task_manager/_search"

Elastic Stack Elasticsearch
1

Hi everyone,
today we faced to this log with level WARN:

{"type": "server", "timestamp": "2021-10-23T05:13:43,815Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch_cluster", "node.name": "elasticsearch", "message": "path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager}", "cluster.uuid": "ryMkKNmhLkwMgAkH", "node.id": "rTw2DEdfgRRT34Nkj7S" , 
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:636) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:357) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:669) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:440) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:289) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:62) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:405) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:745) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1290) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1399) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1373) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:720) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.12.0.jar:7.12.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: org.elasticsearch.action.NoShardAvailableActionException: [elasticsearch][192.168.64.2:9300][indices:data/read/search[phase/query]]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:473) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:422) [elasticsearch-7.12.0.jar:7.12.0]",
"... 18 more"] }
{"type": "server", "timestamp": "2021-10-23T05:13:43,837Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch_cluster", "node.name": "elasticsearch", "message": "path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager}", "cluster.uuid": "ryMkKNmhLkwMgAkH", "node.id": "rTw2DEdfgRRT34Nkj7S" , 
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:636) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:357) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:669) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:440) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:289) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:62) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:405) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:745) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1290) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1399) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1373) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:720) [elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.12.0.jar:7.12.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: org.elasticsearch.action.NoShardAvailableActionException: [elasticsearch][192.168.64.2:9300][indices:data/read/search[phase/query]]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:473) ~[elasticsearch-7.12.0.jar:7.12.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:422) [elasticsearch-7.12.0.jar:7.12.0]",
"... 18 more"] }

we launched single-node Elasticsearch:7.12.0 and kibana:7.12.0 with Docker and following configs:

environment:
      - node.name=elasticsearch
      - discovery.type=single-node
      - cluster.name=elasticsearch_cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms31G -Xmx31G"
      - http.max_content_length=1536mb
      - cluster.routing.allocation.disk.threshold_enabled=false

and indexing rate is very high and for a day, size of index became over than 600 GB with at least 40 millions documents.
What is the reason for this WARN log?
I want to thank you in advance.

2

I found the problem. These logs were recorded at the start of the Elasticsearch node and at that time The shards themselves are still starting up at this point, which can take some time.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.