Yellow cluster....why?


#1

Hi all,
my kibana monitoring says:

Elasticsearch cluster status is yellow. Allocate missing replica shards.

But all shards seems to be allocated:

GET _cat/shards return taht all shards are started. Nothing else.
How can i troubleshoot the problem????


(Val Crettaz) #2

What do you see when running the following commands?

curl localhost:9200/_cat/indices?v
curl localhost:9200/_cat/shards?v


#3

All indices started with 1p 1r 0r 0p
All shards green with 2 pri and 1 rep, except for someone that has 1 pri 1 rep.


(Val Crettaz) #4

Ok that means you have two shards per index with one replica each.
And how many nodes are there in your cluster?


#5

3 nodes..


(Val Crettaz) #6

What do you get when running (please provide the full response)

curl localhost:9200/_cluster/allocation/explain


#7

root@elastic01-test:~# curl localhost:9200/_cluster/allocation/explain
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node3][172.16.0.6:9300][cluster:monitor/allocation/explain]"}],"type":"illegal_argument_exception","reason":"unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"},"status":400}


(Val Crettaz) #8

Seems normal indeed. Can you restart/refresh Kibana, just in case?


#9

Same warning..:frowning:

Elasticsearch cluster status is yellow. Allocate missing replica shards.
Last checked December 29, 2017 1:00:32 AM (since 5 days 15 hrs 16 min ago)

Also, in elastic logs in the indicated timestamp i have no error!!


(Val Crettaz) #10

Are all nodes data nodes which can host data? or some of them are master nodes?


#11

all data node and 1 of three is master node. All nodes are master node eligibles


(Val Crettaz) #12

Do you happen to have XPack installed and system indices starting with a dot, i.e. .monitoring-xyz?


#13

Yes i have:

green open .monitoring-es-6-2018.01.01 ejMYeHmaRSuvYYzYluMNPQ 2 1 1294130 11465 1.5gb 817.7mb
green open .monitoring-alerts-6 b1pbQodrQYSh-WEoKfqyhg 1 1 2 0 26kb 13kb
green open .monitoring-es-6-2018.01.03 9RJH4LRWTfa5UO_77e3Z5Q 2 1 669263 10168 1gb 495.7mb
green open .monitoring-es-6-2017.12.31 swfcIHxVROSU99dL0t6cfw 2 1 1304559 7312 1.5gb 811mb
green open .monitoring-logstash-6-2018.01.01 tDk930glRGm1wbmxqrQ8jA 2 1 1603780 0 681.4mb 343.3mb
green open .monitoring-logstash-6-2018.01.02 y0uGT989RkSnehiMM-sY4Q 2 1 1600912 0 669.1mb 337mb
green open .monitoring-logstash-6-2017.12.30 fm9U7HYnQOOrLqywU4euDA 2 1 1600148 0 682.4mb 343.7mb
green open .monitoring-kibana-6-2018.01.03 whu_7bFTRI6W15V5gCaJbw 2 1 2614 0 1.4mb 772.6kb
green open .monitoring-es-6-2018.01.02 ZRU2QCO3RxmMnrXxYgHeJg 2 1 1079885 13560 1.3gb 690.1mb
green open .monitoring-es-6-2017.12.30 8QAagXYkT7SWyDglH1TMLA 2 1 1142098 99461 1.4gb 722.3mb
green open .monitoring-logstash-6-2018.01.03 oLvlg5eRTLyNGMH-b6y4Rw 2 1 999777 0 419.7mb 207.8mb
green open .monitoring-logstash-6-2017.12.31 VsJFlKQ8RP2P7VJDNCSPiA 2 1 1599258 0 678.6mb 335.8mb


(Val Crettaz) #14

I still would like to see the full output of the _cat/shards?v call.


#15

Uploaded:
https://drive.google.com/file/d/1fPNYK9kY-gtPp5Gb412dEJXbGgyfFdnT/view?usp=sharing


(Val Crettaz) #16

Can you screenshot what you see in Kibana under "Monitoring > Elasticsearch > Indices" (eventually check the "Show system indices" box).
Do you see any yellow index? Do you see values > 0 in the last column named "Unassigned shards"?


#17

Hi have 20 pages and more.....
but, no yellow index, all is green and no unassigned shards!!


(Val Crettaz) #18

One more, can you show the result you get from _cluster/health?


#19

{"cluster_name":"elk","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":473,"active_shards":946,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}


(Val Crettaz) #20

Interesting one... And of course Kibana and ES have exactly the same version?