Yellow cluster....why?

Elastic Stack Elasticsearch
1

Hi all,
my kibana monitoring says:

Elasticsearch cluster status is yellow. Allocate missing replica shards.

But all shards seems to be allocated:

GET _cat/shards return taht all shards are started. Nothing else.
How can i troubleshoot the problem????

2

What do you see when running the following commands?

curl localhost:9200/_cat/indices?v
curl localhost:9200/_cat/shards?v

3

All indices started with 1p 1r 0r 0p
All shards green with 2 pri and 1 rep, except for someone that has 1 pri 1 rep.

4

Ok that means you have two shards per index with one replica each.
And how many nodes are there in your cluster?

5

3 nodes..

6

What do you get when running (please provide the full response)

curl localhost:9200/_cluster/allocation/explain

7

root@elastic01-test:~# curl localhost:9200/_cluster/allocation/explain
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node3][172.16.0.6:9300][cluster:monitor/allocation/explain]"}],"type":"illegal_argument_exception","reason":"unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"},"status":400}

8

Seems normal indeed. Can you restart/refresh Kibana, just in case?

9

Same warning..:frowning:

Elasticsearch cluster status is yellow. Allocate missing replica shards.
Last checked December 29, 2017 1:00:32 AM (since 5 days 15 hrs 16 min ago)

Also, in elastic logs in the indicated timestamp i have no error!!

10

Are all nodes data nodes which can host data? or some of them are master nodes?

11

all data node and 1 of three is master node. All nodes are master node eligibles

12

Do you happen to have XPack installed and system indices starting with a dot, i.e. .monitoring-xyz?

13

Yes i have:

green open .monitoring-es-6-2018.01.01 ejMYeHmaRSuvYYzYluMNPQ 2 1 1294130 11465 1.5gb 817.7mb
green open .monitoring-alerts-6 b1pbQodrQYSh-WEoKfqyhg 1 1 2 0 26kb 13kb
green open .monitoring-es-6-2018.01.03 9RJH4LRWTfa5UO_77e3Z5Q 2 1 669263 10168 1gb 495.7mb
green open .monitoring-es-6-2017.12.31 swfcIHxVROSU99dL0t6cfw 2 1 1304559 7312 1.5gb 811mb
green open .monitoring-logstash-6-2018.01.01 tDk930glRGm1wbmxqrQ8jA 2 1 1603780 0 681.4mb 343.3mb
green open .monitoring-logstash-6-2018.01.02 y0uGT989RkSnehiMM-sY4Q 2 1 1600912 0 669.1mb 337mb
green open .monitoring-logstash-6-2017.12.30 fm9U7HYnQOOrLqywU4euDA 2 1 1600148 0 682.4mb 343.7mb
green open .monitoring-kibana-6-2018.01.03 whu_7bFTRI6W15V5gCaJbw 2 1 2614 0 1.4mb 772.6kb
green open .monitoring-es-6-2018.01.02 ZRU2QCO3RxmMnrXxYgHeJg 2 1 1079885 13560 1.3gb 690.1mb
green open .monitoring-es-6-2017.12.30 8QAagXYkT7SWyDglH1TMLA 2 1 1142098 99461 1.4gb 722.3mb
green open .monitoring-logstash-6-2018.01.03 oLvlg5eRTLyNGMH-b6y4Rw 2 1 999777 0 419.7mb 207.8mb
green open .monitoring-logstash-6-2017.12.31 VsJFlKQ8RP2P7VJDNCSPiA 2 1 1599258 0 678.6mb 335.8mb

14

I still would like to see the full output of the _cat/shards?v call.

15

Uploaded:
https://drive.google.com/file/d/1fPNYK9kY-gtPp5Gb412dEJXbGgyfFdnT/view?usp=sharing

16

Can you screenshot what you see in Kibana under "Monitoring > Elasticsearch > Indices" (eventually check the "Show system indices" box).
Do you see any yellow index? Do you see values > 0 in the last column named "Unassigned shards"?

17

Hi have 20 pages and more.....
but, no yellow index, all is green and no unassigned shards!!

18

One more, can you show the result you get from _cluster/health?

19

{"cluster_name":"elk","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":473,"active_shards":946,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

20

Interesting one... And of course Kibana and ES have exactly the same version?