All shards failed-yellow status

Elastic Stack Elasticsearch
1

ELK: version 6.6.2

Hi,

my elasticsearch stopped working after couple weeks. Im monitoring cca 10 tomcat logs ondifferent servers. I run ELK with Logstash on same server(SINGLE NODE). In log there is still repeating:

[2020-01-14T00:00:26,348][WARN ][r.suppressed ] [wcnXOKN] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

My memory:

[root@HCgiKram bin]# free -g
total used free shared buff/cache available
Mem: 7 5 0 0 2 1
Swap: 3 0 2

In JVM option I have set -Xmx3g -Xms3g

Output from health api:

{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 746,
"active_shards" : 746,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 745,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.033534540576795
}

Kibana wont even start, log error:

FATAL [search_phase_execution_exception] all shards failed :: {"path":"/.kibana/doc/_count","query":{},"body":"{"query":{"bool":{"should":[{"bool":{"must":[{"exists":{"field":"index-pattern"}},{"bool":{"must_not":{"term":{"migrationVersion.index-pattern":"6.5.0"}}}}]}}]}}}","statusCode":503,"response":"{"error":{"root_cause":,"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":
},"status":503}"}

I read that yellow status is normal on single node, but why is kibana not starting? Please can anybody help?

2

Please anybody???

3

Read this and specifically the "Also be patient" part.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

4

Please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

Don't use the citation icon but the code format icon.

5

You probably can see something in elasticsearch logs. Could you share the logs?

With 3gb of RAM, you should probably not have more than 60 shards on the node.
Here you have 746 shards which is a way too much.

The yellow status is expected as you probably have indices with more than 0 replica. As you have only one data node, replicas can't be allocated.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.