Elasticsearch node fails after kibana startup .action.search.SearchPhaseExecutionException: all shards failed

Elastic Stack Elasticsearch
1

I have 2 nodes

{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 3,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Once i start Kibana, I receive 2 different message from master nodes.

Just want to understand if this is as per design or something is wrong

{"type": "server", "timestamp": "2019-09-16T12:13:23,505+0000", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "cluster.uuid": "wbPF17P8SVKoOxbu9ESiWA", "node.id": "58CjS3W3R1m4FYwnPfPfOw",  "message": "Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...])."  }
^R


{"type": "server", "timestamp": "2019-09-16T12:09:57,555+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "wbPF17P8SVKoOxbu9ESiWA", "node.id": "g3MOxW7_TnenDyigGrWEyw",  "message": "All shards failed for phase: [query]"  }
{"type": "server", "timestamp": "2019-09-16T12:09:57,556+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "wbPF17P8SVKoOxbu9ESiWA", "node.id": "g3MOxW7_TnenDyigGrWEyw",  "message": "path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager}" , 
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",

After that I see elasticsearch node went down

k exec -ti elasticsearch-master-1 bash -n logging
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "d2b1bf3399cb5c10a41215bf82c304d660e69ee6bef8811df4131ef0a614d8b8": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown

but pod looks like still running, very strange

NAME                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0   1/1     Running   0          45m
elasticsearch-master-1   1/1     Running   0          45m
kibana-788597cf8-ctqbb   1/1     Running   0          21m
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.