[error] All shards failed after migration to 6.7.1

Elastic Stack Elasticsearch
1

Hi
I have done this morning an upgrade of my elk stack, to 6.3.0 to 6.7.0

here the steps

disable shards allocation
update elasticserach

when the "active_shards_percent_as_number" was 50% (yellow state)

update logstash
update kibana

then I have problems :frowning:

I have an error when I see the logs of kibana, I was thinking the problem was kibana but I think not.
In elasticsearch I have this :

[2019-04-15T11:39:55,167][WARN ][r.suppressed ] [prod-master-1] path: /.kibana_task_manager/_doc/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, type=_doc}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

the status is red

"cluster_name" : "logging-prod",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1507,
"active_shards" : 1507,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1511,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 49.93373094764745

I have 2 indices in red : ".kibana_task_manager" and ".kibana_7"

How I can fix this ?

Do I have to enable again shard allocation ?

2

According to the upgrade docs you should still allow primary shard allocation:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

However if you have completely disabled shard allocation then these two new indices will have unassigned primaries, leading to a red cluster health. Try and re-enable allocation (at least, set it to primaries) to allow these primaries to be assigned.

3

you saved my life :smile:

I have done this

curl -X PUT "127.0.0.1:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster.routing.allocation.enable": "primaries"
}
}
'

the status was yellow, then no more error in elasticsearch.
Then I have restarted Kibana and I had an error with index .kibana-7 so I have deleted it and it's OK now.

what's the difference between none and primary on the allocation shard ?

Thank you for your answer

4

When you create a new index, Elasticsearch has to allocate the shards to nodes before you can write to them. If you set cluster.routing.allocation.enable: none then Elasticsearch will never allocate any shards, whereas if you set cluster.routing.allocation.enable: primaries then Elasticsearch will allocate primary shards but not replicas. You cannot write to a shard until at least its primary is allocated, so the cluster health reports red when it is not possible to allocate its primary.

It's possible the docs are also useful to you.

5

ok thank you !

I have read the latest doc, it's written "none". Next time, I'll be more aware !!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.