Yellow cluster....why?


Hi all,
my kibana monitoring says:

Elasticsearch cluster status is yellow. Allocate missing replica shards.

But all shards seems to be allocated:

GET _cat/shards return taht all shards are started. Nothing else.
How can i troubleshoot the problem????

(Val Crettaz) #2

What do you see when running the following commands?

curl localhost:9200/_cat/indices?v
curl localhost:9200/_cat/shards?v


All indices started with 1p 1r 0r 0p
All shards green with 2 pri and 1 rep, except for someone that has 1 pri 1 rep.

(Val Crettaz) #4

Ok that means you have two shards per index with one replica each.
And how many nodes are there in your cluster?


3 nodes..

(Val Crettaz) #6

What do you get when running (please provide the full response)

curl localhost:9200/_cluster/allocation/explain


root@elastic01-test:~# curl localhost:9200/_cluster/allocation/explain
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node3][][cluster:monitor/allocation/explain]"}],"type":"illegal_argument_exception","reason":"unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"},"status":400}

(Val Crettaz) #8

Seems normal indeed. Can you restart/refresh Kibana, just in case?


Same warning..:frowning:

Elasticsearch cluster status is yellow. Allocate missing replica shards.
Last checked December 29, 2017 1:00:32 AM (since 5 days 15 hrs 16 min ago)

Also, in elastic logs in the indicated timestamp i have no error!!

(Val Crettaz) #10

Are all nodes data nodes which can host data? or some of them are master nodes?


all data node and 1 of three is master node. All nodes are master node eligibles

(Val Crettaz) #12

Do you happen to have XPack installed and system indices starting with a dot, i.e. .monitoring-xyz?


Yes i have:

green open .monitoring-es-6-2018.01.01 ejMYeHmaRSuvYYzYluMNPQ 2 1 1294130 11465 1.5gb 817.7mb
green open .monitoring-alerts-6 b1pbQodrQYSh-WEoKfqyhg 1 1 2 0 26kb 13kb
green open .monitoring-es-6-2018.01.03 9RJH4LRWTfa5UO_77e3Z5Q 2 1 669263 10168 1gb 495.7mb
green open .monitoring-es-6-2017.12.31 swfcIHxVROSU99dL0t6cfw 2 1 1304559 7312 1.5gb 811mb
green open .monitoring-logstash-6-2018.01.01 tDk930glRGm1wbmxqrQ8jA 2 1 1603780 0 681.4mb 343.3mb
green open .monitoring-logstash-6-2018.01.02 y0uGT989RkSnehiMM-sY4Q 2 1 1600912 0 669.1mb 337mb
green open .monitoring-logstash-6-2017.12.30 fm9U7HYnQOOrLqywU4euDA 2 1 1600148 0 682.4mb 343.7mb
green open .monitoring-kibana-6-2018.01.03 whu_7bFTRI6W15V5gCaJbw 2 1 2614 0 1.4mb 772.6kb
green open .monitoring-es-6-2018.01.02 ZRU2QCO3RxmMnrXxYgHeJg 2 1 1079885 13560 1.3gb 690.1mb
green open .monitoring-es-6-2017.12.30 8QAagXYkT7SWyDglH1TMLA 2 1 1142098 99461 1.4gb 722.3mb
green open .monitoring-logstash-6-2018.01.03 oLvlg5eRTLyNGMH-b6y4Rw 2 1 999777 0 419.7mb 207.8mb
green open .monitoring-logstash-6-2017.12.31 VsJFlKQ8RP2P7VJDNCSPiA 2 1 1599258 0 678.6mb 335.8mb

(Val Crettaz) #14

I still would like to see the full output of the _cat/shards?v call.



(Val Crettaz) #16

Can you screenshot what you see in Kibana under "Monitoring > Elasticsearch > Indices" (eventually check the "Show system indices" box).
Do you see any yellow index? Do you see values > 0 in the last column named "Unassigned shards"?


Hi have 20 pages and more.....
but, no yellow index, all is green and no unassigned shards!!

(Val Crettaz) #18

One more, can you show the result you get from _cluster/health?



(Val Crettaz) #20

Interesting one... And of course Kibana and ES have exactly the same version?