Hello all,
I had some problems with the upgrade of Elasticsearch 6.5.3 to 6.6.2.
After the update the connection between kibana and elasticsearch was lost.
I was prompted with error 500 (internal server error).
Was forced to do a rollback to 6.5.3.
Now I can't allocate today's shards, i got the following error on elasticasearch:
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy"
On the logstash instance the error is:
[logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[####-2019.03.19][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[####-2019.03.19][0]] containing [12] requests]"})
Besides the data shards all the monitoring ones are unassigned.
I disabled the allocation before the update and enabled after.
Also stopped the non-essential indexing and performed a synced flush
Some more information:
GET _cluster/health
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 352,
"active_shards" : 352,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 10,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 97.23756906077348
}
Any idea on how to sort this?