ECK 1.5 cluster failed after restart nodes by "this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node"

[rueian]

Hi, I am testing Elasticsearch cluster 7.12.1 with 3 nodes managed by ECK 1.5.

Every thing works fine until I restart all GKE nodes.

Steps:

1. create GKE cluster
2. apply eck_all_in_one_1.5.0.yaml
3. apply my elasticsearch crd resource, and elasticsearch cluster successfully starts
4. resize GKE node pool to zero, and resize back
5. the elasticsearch pods restarts but continuously print out "this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node"

I have tried recreating the cluster by delete and apply my crd resource again, but that also deleted all my data.
Is there any other way to restart a elasticsearch cluster managed by ECK after all nodes shutted down?

[sebgl (Sebastien Guilloux)]

This looks like your Elasticsearch Pods have started with an empty data volume. Is there a chance your PersistentVolumes or their content was deleted?

Could you post your elasticsearch resource manifest (yml)?
I'm suspecting something might be wrong with your volume claims.

If using network-attached PersistentVolumes, Pods should be reattached to the existing volumes, you shouldn't see that error.